MAIN README
Author: IanB - ian(at)i-asm.com
|
It has a simple design principle. While yDecode is running, binary files to be decoded from a Usenet source such as a newsreader should be saved into a single input folder. This should be achieved automatically by means of settings in the newsreader if possible. If the necessary input and output folders are correctly specified, yDecode will start decoding every yEnc or UUE file that arrives in that folder automatically, else it will wait for instructions on where to process from and to and whether to begin. When it finishes a queue of files, it waits five seconds before checking the folder for a new queue so as not to actively consume system resources.
But binary data downloaded from Usenet is prone to errors in transmission, loss of parts etc., and usually requires multiple utilities to verify it fully, repair and then piece it all back together into the original file(s). yDecode doesn't just act as a "dumb" decoder, therefore. It examines everything it decodes to see if it is a verification file (CRC/CSV/SFV/PAR/PAR2) for all other files that have already been decoded or may arrive afterwards. The verification data is then used by yDecode to check and if necessary repair existing and incoming files if possible.
However, where yDecode's verification differs from QuickPar and other utilities is that no user intervention is required! Instead of manually selecting a specific dataset and checking the files, then joining the split parts if necessary and deleting the unneeded parts afterwards, yDecode automates all these tiresome processes entirely! Any split files are recognised and joined and any unnecessary extra files are all deleted for you if the joins/repairs completed without error. It always errs on the side of caution, however, and for safety it verifies rigorously as it goes, using the best verification source available (PAR2 by preference).
It handles multiple "open" verification sets, so data can arrive from the newsreader source in any order. Its status window shows what verification sets are currently being processed. It also allows users to easily create comprehensive filter rules so that downloaded files can be routed automatically to any number of folders depending on their type and their newsgroup source in any combination!
Therefore, once the full set of source file messages along with some CRC/PAR/PAR2 verification have been selected for download in the newsreader program, the user should need do nothing except wait for the newsreader to complete the download, then access the final complete saved file(s), fully verified, routed and joined if required, with all intermediate files safely cleaned up, unless the required PAR/PAR2/SFV/CRC/CSV verification has not been provided by the original poster.
Finally, with appropriate option settings, all unnecessary source files will be automatically deleted (this is the default action), only if decoding/processing is completed and verified without any errors however, else they are simply renamed with a .DONE extension. Completed split files and verification files can also be removed automatically as required.
For those unfamiliar with NewsPlex, it allows the use of multiple news-servers to be transparently combined into one single "virtual" news-server which can then be accessed by your local preferred newsreader program. NewsPlex handles all the direct server requests in the background and tracks which message articles are available and where. This means articles available on at least one server known to NewsPlex will be visible and accessible in one place without the user having to know exactly which server they are on to retrieve them.
Many of yDecode's best features require the use of this powerful software as a source of files to decode. It will check to see if it is in a NewsPlex installation, and will automatically set the correct NewsPlex \async subdirectory as the input folder if it is. It reads a number of settings from the etc\newsplex.ini file (for NewsPlex 4.0+) to enable automatic connection via NNTP link. It is also simple to start and stop NewsPlex remotely at any time or automatically in sync with yDecode.
It also requires the use of one of NewsPlex's best features: asynchronous article retrieval. This allows binary files to be removed entirely from the purview of the newsreader software to be handled more effectively in the background, in this case by yDecode, thus freeing it up for other tasks, such as reading text-only messages (and handling mime-encodes!). Users should appreciate the difference if they have been used to, say, Agent's "freezing" when dealing with large numbers of large binary downloads!
Most importantly, if there is an active NNTP link to a running instance of NewsPlex, yDecode can automatically request any repair files it needs from Usenet! For instance, if it calculates that 5 extra repair blocks are needed to complete a PAR2-verified set, then it can silently request them through NewsPlex and use them when downloaded without any user intervention required, if this option is allowed. This is yDecode's most unique and powerful feature.
The only answer for a fully automatic solution was to build my own yEnc decoder, but this has allowed me to build in extensive additional features to what was originally a fairly unambitious project!
Those include UUE decoding, automatic operation, completed file deletion, output file folder routing, file verification and splitfile handling, not to mention NewsPlex integration. My proudest achievement, with the very generous assistance of Peter Clements, one of the authors of the PAR2 specification, is that I've also managed to integrate PAR and PAR2 repair.
I believe that the combination of NewsPlex and yDecode is one of the most powerful automatic newsgroup binary downloading solutions available, irrespective of the chosen newsreader front end. Full verification and PAR/PAR2 repair is now integrated, so the only user intervention required is to find replacement/repair files manually if NewsPlex is not running.
With a working NNTP feedback channel to NewsPlex, PAR and PAR2 repair files can be automatically queued for download without any user intervention and will be automatically used to repair files if necessary, cleaning up partial downloads and leaving only good, verified data. Replacements for corrupt files which cannot be repaired (CRC-type verification) are also requested if available.
Full control over NewsPlex itself and its async queue is available through an integrated user Telnet link. While this is currently restricted to direct user text input only, requiring full user knowledge of all options, some integrated help along with useful one-click command settings, and differently filtered query/reporting options, may be implemented in future.
Along the way I've learnt a great deal about Visual Basic, writing for the Windows API and good class-based software design, but most importantly for users I've also taught myself Assembler programming. yDecode 4.0 is the first release that uses some of the new Assembler code I have written specially to handle the most speed-critical processing (the verification parsing and repair in particular) and I am in the process of rewriting the majority of yDecode into hand-coded Assembler so future releases will be even more rock-solid, fast and less reliant on inefficient and flaky VB routines.
Even in this new interim version, yDecode is approaching the speed and efficiency of other mainstream standalone PAR/PAR2 utilities, that do much less. It is only going to get better! Look out for upgraded versions soon.
yDecode really is an indispensible Usenet tool. I hope you find it useful.
It is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. You can use it at your own risk, but you are encouraged to report any problems you may encounter to the author so that yDecode can be improved.
You are free to copy this software ONLY if you include this README and all manual files in full and unaltered along with all other accompanying files in the package. You may NOT charge anyone for a copy of this software other than a small copying fee. You may NOT include or combine this software with any commercial software without the written consent of the author.
If you find yDecode useful, please consider donating to the author towards the time he has spent creating it, and to encourage further development and upgrades. There is a Paypal donation link at the top of this page.
For info on yDec or .dec files, see http://www.yenc.org/ydec.htm
For UUE specifications (best reference I've found), see uuencode.5 at http://www.opensource.apple.com
For info on NewsPlex, see the Unofficial NewsPlex FAQ Page and the NewsPlex Yahoo! discussion group.
For info on PAR and PAR2, see the PARchive website http://parchive.sourceforge.net
to Peter Clements for incredibly patient PAR2 support and getting me excited about Galois Field arithmetic and the possibility for freeware Usenet utilities: http://www.quickpar.org.uk
to Emiliano Scavuzzo for great CSocketMaster class VB code that allowed me to remove dependency on Winsock: http://www.geocities.com/anshoku
to members of the NewsPlex Yahoo! discussion group for testing, suggestions and bug reporting