The Traders' Den  

  The Traders' Den > Where we go to learn ..... > Technobabble
 

Notices

Technobabble Post your general Need for Help questions here.
Lossy or Lossless?
Moderators

Reply
 
Thread Tools
  #1  
Old 2022-10-26, 04:33 PM
frank carson frank carson is online now
71.76 TB/107.34 TB/1.50
 
Join Date: Sep 2018
Automating Reseeding - Method 1 - Locate Filesets and Reload Torrents

= The Challenge

You want to reseed torrents efficiently but the overhead of manually tracking down the fileset for a torrent and reloading it in your client while pointing to the existing fileset is overwhelming.

-= A Solution

The "Locate Fileset By Torrent" task will take a source directory of torrent files and search a list of SearchPath(s) in your archive for the fileset(s) that back those torrent files and spit out a (customizable) report linking the torrent file to the fileset. If you customize this report to generate the command to load each torrent into your client backed by the existing fileset you can automate this part of the reseeding process.

Examples are included for Transmission using the RPC api but any client with a similar API should also be scriptable.

-= Prerequisites

Your downloaded filesets have the same folder name as the "name" in the torrent file.

- This is (currently) required since we initially match the fileset folder name against the "name" attribute in the torrent. Additionally a "Directory Content Hash" of the fileset contents and a "Torrent Content Hash" of the torrent contents is used to verify the contents of the fileset folder match up with the expected contents of the torrent file (this detects incomplete filesets etc.) Ref: https://github.com/ossprj/ossprj-com...orycontenthash AND https://github.com/ossprj/ossprj-com...entcontenthash

- A future enhancement will be the option to ignore the folder names and match solely on the "Directory Content Hash" and "Torrent Content Hash", which would allow you to rename your filesets any way you please.

Your filesets must be organized in some type of consistent hierarchy.

- This is required so that the fileset search algorithm understands which directories are to be considered filesets and which are to be ignored as organizational only. The SearchPath(s) specification allows for the specification of a base directory and search depth and the "Locate Fileset By Torrent" task allows for the inclusion of multiple SearchPath(s), however the more consistent your folder organization, the fewer and more compact SearchPath specifications you can use. Ref: https://github.com/ossprj/ossprj-com...ile#searchpath

- For example, if your filesets are organized into a hierarchy like: /Archive/Artist/Year/Fileset then you could use a SearchPath with "basePath" of "/Archive" and a "searchDepth" of 2 to find all your filesets correctly. However, if only some of your Artist(s) have the filesets organized by year and some have the filesets directly under the Artist then the SearchPath above will not correctly find all your filesets. The more complex your organizational hierarchy the more SearchPath(s) will be required to discover all your filesets.

-= Best Practices

If possible, mount your archive read-only and use that read-only path as the fileset sources. This way any erroneously loaded torrents which may point to older/incomplete filesets won't corrupt your archival filesets.

Use a consistent hierarchy to organize your filesets. Your actual hierachy doesn't really matter as much as applying it consistently. The more consistent your organizational hierarchy the easier/simpler it is to write SearchPath definitions that drive discovery of your filesets.

-= Limitations

The "Locate Fileset By Torrent" task only recognizes directory based filesets at the moment.
- I have a refactoring to handle single files implemented but not fully tested and committed.

-= The Process

Download all the torrents for the filesets you wish to locate into the path configured via "torrentsDirectory"
Configure your List of SearchPaths to point to your archived filesets.
Run the task to generate your report/script.

-= Setup

Requires a Java JRE/JDK 8 *minimum* be installed to run.

All code is freely available on github so you can see what you are running and build it yourself if you want.
Current release is from the "release/1.0.0" tag on all repos
Build requires JDK 8 min and Maven 3.x
Build order is:
- https://github.com/ossprj/ossprj-poms-base
- https://github.com/ossprj/ossprj-poms
- https://github.com/ossprj/ossprj-commons
- https://github.com/ossprj/ossprj-torrent-tasks

See docs for usage details:
https://github.com/ossprj/ossprj-tor...t-tasks-locate

Get the release zip here:
https://github.com/ossprj/ossprj-tor...elease%2F1.0.0

Choose/Create a base directory to work from
- In this example we use "/home/user/TorrentTasks"

Create the directory structure below and unzip the release zip into that directory
- The release zip contains the jars and yml config files.
- My LLegs-Reseed.sh contents are below. Run the same steps but with slightly different syntax for Windows
- Torrents.Reseed is a required directory (contains torrent files for torrents you wish to locate)
- Other directories are optional (Torrents.Incomplete / Torrents.Missing) and are only required if you set the appropriate config parameter (see FileOutput / HandleMissingAndIncomplete configurations)

Run this (in the base directory) to double check you have things setup correctly. All it does is search your archive for matching torrents and then spit out some info to the console
java -Dspring.profiles.active=ConsoleOutput -jar ossprj-torrent-tasks-locate-1.0.0.jar

The example configurations have additional debug level logging enabled but commented out.
Uncomment the line "com.ossprj.torrent.task: DEBUG" if you wish to see some additional debugging info in the console.
Turning this down to DEBUG will show additional details about
- the fully resolved configuration
- the discovered List of filesets at runtime (may be helpful in building your SearchPath(s))

Once you are comfortable its discovering your candidate filesets properly (using the ConsoleOutput config above) you can move onto one of the other configurations that generates the script like
java -Dspring.profiles.active=FileOutput -jar ossprj-torrent-tasks-locate-1.0.0.jar
OR one that will generate the script and move torrents with Incomplete/Missing filesets to a separate directory
java -Dspring.profiles.active=HandleMissingAndIncomplete -jar ossprj-torrent-tasks-locate-1.0.0.jar


-= /home/user/TorrentTasks (Directory Layout)

-rw-rw-r-- 1 user user 1173 Feb 11 14:09 application-ConsoleOutput.yml
-rw-rw-r-- 1 user user 1326 Feb 11 14:10 application-FileOutput.yml
-rw-rw-r-- 1 user user 1781 Feb 11 14:34 application-HandleMissingAndIncomplete.yml
-rwxrwxr-- 1 user user 622 May 6 02:05 LLegs-Reseed.sh
-rw-rw-r-- 1 user user 15842358 May 6 02:07 ossprj-torrent-tasks-locate-1.0.0.jar
drwxrwxr-x 2 user user 4096 May 5 11:47 Torrents.Incomplete
drwxrwxr-x 2 user user 4096 May 5 00:26 Torrents.Missing
drwxrwxr-x 2 user user 12288 May 6 01:52 Torrents.Reseed


-= LLegs-Reseed.sh (Use as-is for *nix, roll your own for Windows)

#!/bin/bash

# Clear out the existing torrents in the reseed dir
rm -fR /home/user/TorrentTasks/Torrents.Reseed/*

# Download a new set of torrents for reseed into "Torrents.Reseed"
# ... You have to do this manually for now

# Generate the shell script to upload torrents to the (Transmission) client
java -Dspring.profiles.active=HandleMissingAndIncomplete -jar ossprj-torrent-tasks-locate-1.0.0.jar

# Load all torrents found for reseed
chmod ug+x /home/user/TorrentTasks/reseed-torrents.sh
/home/user/TorrentTasks/reseed-torrents.sh
Reply With Quote Reply with Nested Quotes
  #2  
Old 2022-12-31, 01:29 PM
xavier242's Avatar
xavier242 xavier242 is offline
TTD Administrator
TTD Staff
3.65 TB/8.38 TB/2.30
 
Join Date: Dec 2008
Location: a warm place
Re: Automating Reseeding - Method 1 - Locate Filesets and Reload Torrents

Seeding Helper v1.28 has been released. It basically does what you are doing but with a friendly efficient GUI. I have reseeded over 100 gigs in a minimal amount of time using it.

Download the 64bit Windows installer here:
https://lgxcreations.com/files/setup...28-win-x64.msi

Clean Virustotal reports:
SeedingHelper.exe
https://www.virustotal.com/gui/file/...478f4f94d5ff9e

setup-seeding-helper-1-28-win-x64.msi
https://www.virustotal.com/gui/file/...871fa503bb1ce6

Full release thread:
http://www.thetradersden.org/forums/...d.php?t=186245

It's free for TTD members.

Reply With Quote Reply with Nested Quotes
Reply

The Traders' Den > Where we go to learn ..... > Technobabble

Similar Threads
Thread Forum Replies Last Post
Reseeding Inactive Torrents - fastcore Technobabble 7 2023-05-30 04:08 AM
Reseeding Tom Petty torrents. - MusicWizard04 Seeding Talk - ISO Requests 1 2022-04-05 07:19 PM
Reseeding Tom Petty torrents. - MusicWizard04 Seeding Talk - ISO Requests 0 2022-04-03 03:04 PM
Reseeding Old Torrents... - Intexity Technobabble 0 2013-08-23 01:19 PM
Would someone mind reseeding these Coldplay torrents for me? Shows enclosed. - Jackal051988 Seeding Talk - ISO Requests 5 2008-04-06 03:16 PM


Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forums


All times are GMT -5. The time now is 12:13 PM.


Powered by: vBulletin, Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright ©2004 - , TheTradersDen.org - All Rights Reserved - Hosted at QuickPacket