view · edit · attach · print · history

The contents of this website are Copyright (c)2005 by Brian Manning <brian at antlinux dot com>. Please do not reuse any of the content on this website without permission from the author.

Related pages:

Streamcast :: Scripts (individual Streamcast scripts) :: Propaganda :: Stream-db :: Documentation

Links on this page:

Documentation :: Support :: Setup :: Metadata (Data about scripts/modules, script versioning, binary requirements, etc.) :: Initialization :: Runtime Operations :: User Interface :: Access Control :: Logging :: Song Queues :: Database Operations :: Support Modules :: Support Scripts :: Packgaging :: Testing :: Reencoding :: Unsorted Todos

Documentation

  • When documenting object methods, make sure you document what the arguments to the object are, as well as what will be returned from the object to the caller.
  • Explain the different sizes of data when streaming, i.e. 128kb/sec is about 16kB/sec, which is ??kB/min and ??MB/hr. See the section on formats and tags
  • add sample config file info into the POD as =begin config/=end config and =begin simpleconfig/=end simpleconfig, so you can parse all of the Propaganda modules and pull out config file info using one of the POD modules
  • document each parameter in streamcast.conf. re-write/document the streamcast.conf file so that the parameters are grouped by which script they apply to
  • set up a sample login session to the admin console, include the default password for streamcast (the one in the crypt hash), also inform of the mkpasswd program in the icecast source.
  • For each command that can be given in the command line administration interface, list what strings can be expected in that command's output; show current should show all of the information for the current song, including most MP3 tags
  • show the default streamcast command in long form, so people know what options are set when running the command all by itself
  • run perltidy on the code, build a list of perltidy options to use when formatting code
  • use Pod::Select to extract sections from the Perl POD documentation for use as part of the interface

Support

  • Add a blurb to the mailing list description on how to get support, then add a support page to the sourceforge page that lists what things would be needed to get problems fixed:
    • icecast/shoutcast version
    • streamcast version
    • distro/version
    • support binaries and their versions (also see the Setup section)

Setup

  • Create a script that creates a monolithic Propaganda script, the user can select which bits and pieces they want to use, and the controller script then copies everything into one big script and removes the use statements, for people who don't want to install everything into their local Perl module tree
  • Document/write a script that obtains debugging information. streamcast version, icecast version, libshout version, Perl version, OS and platform, support binaries installed and their versions, etc. Maybe turn this script into an installer
    • Installer could ask questions of the user using UI::Dialog...
      • include a copy of UI::Dialog with the Streamcast source
        • Write a UI::Dialog module for CocoaDialog, a dialog-type clone for Mac-zen
      • recommend the Debian Perl packager (dh-make-perl) in the docs of Streamcast
      • Mysql username/password/database name
      • streaming server username and password, with link to Live365
      • It could then generate the SQL files needed for the install, with custom database information as gathered by the installer script
  • Write the info gathering/installation tool using Perl's UI classes, including a copy of the most basic classes with the installer if needed
  • Create an inittab startup script for stream*, which will cause init to automatically restart stream* should it die. The script will also su to run streamcast as a non-priviledged user. Possibly try to log errors if something does go wrong, or have the enduser run stream* by itself from the command line, so if it does die, the error message stays on the screen
    • Start script will verify Icecast is up and running before starting streamcast
  • Run with perl -T, and untaint stuff read in from the admin sockets (both local and network). For now, data on the streamcast control socket will be trusted

Metadata (data about scripts/modules and their requirements)

  • create a template for Streamcast modules that contains mandatory methods
    • new() - creates the object, initializes any variables, returns the object to the caller
    • identify() - returns the name and version number (?) to the caller
    • sample_config() - returns a commented or non-commented block of text that has all of the config parameters that the module will accept
    • register() - registers callbacks with the main script
  • parse the CVS $Id$ tag from each file that is described in order to obtain their version information, author, etc.
  • Self describing modules, describe their dependencies:
    • $module->new->dependencies(); would dump the modules that $module needs
  • Create a --describe option that creates objects from each plugin module, and then calls their describe() method, which in turn runs perlpod on that module and outputs the results to the user; this way, the user can see what's installed and what each installed module does
    • --modules help calls the module_description() method for each module; if the module has description text, then return the text so that the caller can output it to the user; if there is no module description text, return undef and the caller can skip that module as far as describing it to the user
  • running --version will silently try to load external modules, and then print out the success/failure of the module loading, as well as the version of the script and the database (if used)
  • When called with the --generate switch, generate a sample streamcast.conf file and send to STDOUT
  • create a config file generator, so that the user can enter in their desired streams and playlist file data, and the configfile generator will create the proper config file, with the config hashes set up the way they need to be set up
  • have an Init/Register/Query function for each new type of object, the checks for external binaries that are needed to manipulate that object can be performed here
  • make the perl database config live in streamcast conf for both stream-db.pl and mp3tosql.pl, so you can change databases in one place

Initialization

  • create a menu population system, which would read menu items from a config file and use it to populate perl-gtk2 or Term::ShellUI? menu systems as needed 
 $menu->add(q(menu item text), $handler_object, q(parent menu item))
 $handler_object->action(q(this is the action);
 sub action{ perform some menu action here }
  • have forking and threading models; the streamcast modules are written to be agnostic between forked servers and threaded servers; the main streamcast script juggles/coordinates global objects between all of the modules
  • Write wrappers around optional Perl modules (Log::Log4perl) that may not be installed by the end user
    • Write the stubs that the missing program would need: $logger = get_logger()
    • Set $logger to undefined if there is no logger object set up in the script
      • (if (defined $logger))
    • wrap the $logger = get_logger() assignment in an eval{} block so if there is no logger, the program continues to run without it. Every time $logger is used however, it will have to be tested for
    • Class::Singleton could be used as a 'one stop shop' for module initialization; an object could be written to handle logging, passing on information to Log4perl if the module was available
  • Use Class::Singleton for handling storage as well; open one object to each list of songs, use the same object throught the scripts.
  • If specified in the config file, use Net::HTTP to query the project page on Sourceforge for the current version of the scripts, and warn the user if they are using an out-of-date version.
  • replace the socket filename in the config file with a path to the socket filename, then generate a socket filename within the script using the PID of the script; this will prevent duplicate sockets; have the socket be cleaned up once stream-db exits
  • build a paramchecker, a hash of streamcast parameters; the script checks for all of the parameters it needs at startup, and for each parameter that is needed that is present, it gets delete()ed from the hash, if there are any keys left over in the hash when all the parameters are checked, then they are join()ed, printed out with an error message, and the program dies
  • On some operating systems assigning to $0 modifies the argument area that the ps program sees. This is more useful as a way of indicating the current program state than it is for hiding the program you're running.
    • Use $0 to set "playlist" and "streamer" for each set of processes that are launched for a stream
  • only show Perl errors when the script is called with --verbose; otherwise, give nice friendly errors
  • add a use lib to load modules based on the $streamcast_modules variable set in the config file
  • Open __FILE__ and parse it for the CVS version; use this as the $VERSION number of that file.
  • Perl Taint mode checks/coding practices

Runtime Operations

  • Add an option to pipe to STDOUT instead of streaming to an Icecast server; this will help with debugging on machines that can't run Icecast (tochka)
  • Allow command aliasing, so calling alias <some command string would bind that command string to the alias specified for it
  • Add a test command, which would print out what the command would do (execute a SQL query, run a system call, whatever) instead of actually performing the command
  • Return values from methods/scripts; decide on a standard for return values from Perl object methods and/or external programs/scripts. Most external programs use the convention where 0 = success, and !0 = failure; what should the object methods do?
  • Search metalanguage
    1. create a search with the search command
    2. the search shows the number of matches
    3. if there are too many matches, then allow the user to add filters to the search with a grep command
    4. show the results a search with the view command
    5. save a search with the save command
    6. clear the current search with the clear command
    7. load a saved search with the load command
    8. list the current search commands with list
    • the search function will also have plain regexp and DOS wildcard modes
  • Write a dump() method for Streamcast objects; the dump() method would dump all of the attributes of the specified object using Data::Dumper; dump() would only be available as a menu choice when DEBUG is enabled
  • If some other process sends streamcast a HUP signal, have streamcast parse the configuration file (if any) again. Re-parsing the config file should also be made available from the control interface as well.
  • Enable throttling for reencoding streams, so that reencoding stops past a certain load for a certain amount of time, say a constant load of 2 for more than 5 minutes.
    • One stream would always be running, but other streams would run or not run depending on the load conditions of the machine
    • not having a throttle setting would most likely cause a denial of service when the system load gets too high
  • Use the demo that comes with Shout.pm to build a super-simple streamer, for all of those people who don't want to hassle with the extra Perl modules (working demo checked into CVS that reads files from STDIN)
  • look into using one socket() to feed multiple encoder forks; see the Socket Perl module for more ideas. Socket is how the streamer and playlist process communicate.
  • use a file socket or FIFO instead of a network socket for the admin interface. Start an admin network listener only when requested by the user/config file
  • Allow the size of the blocks that are read from the media file to be adjustable; default currently is 3k blocks
  • make split streams, a low bitrate stream for reencoding/broadcast, and a high-bitrate stream for internal networks
  • See about making the socket file more unique, so you can run multiple versions of streamcast at once
  • Have the streamer script compare the MP3 title tag against the song's filename, and if the title tag is shorter than the filename, use the filename (minus ".mp3") instead of the title tag when sending the metadata to the server or displaying it on the screen.
  • Have the streamcasters output filenames instead of reading binary files; you could then chain streamcasters to IceS, since IceS will accept filenames as input, but it lacks database support. http://www.icecast.org/docs/ices-2.0.0
    • Make Shout.pm modular, so that if you plan on chaining streamcasters to IceS, you don't have to have Shout.pm installed to satisfy the require at the top of the streamcasters scripts.
  • Create a script output "protocol" flag, so that additions can be made to the script output and clients can choose what version of the script output they want to see. For instance, create a new version of the protocol that feeds either the ID3 tag or the filename information based on what source of information about the file contains more information (i.e. song name or artist name is over 32 characters, which would end up being truncated in an ID3 tag) and then sets a flag in the script output that lets the downstream program know what source was used for the song information
  • Make show songs pause between pages on socket interface
  • YP directory server support?
  • add VBR encoding support
  • Add ability to see song progress (CGI scripts already support this)
  • List songs alphabetically and other sorting options
  • Limit on number of requests/hr based on cookies
  • store the current song in a hash, and update the hash when a new song gets played; this will allow stream-db to display the current song (sho cur) without doing a database lookup, which really isn't needed, as we did a lookup to find out which song to play in the first place.
  • make a play timer/counter, if the streamer requests too many songs in a short amount of time, then everybody dies
  • remove (all|###) request(s)
  • show -> songs - limit by a certain number, if mp3main is over 30 songs or so
  • fix uptime counter
  • make sure that the program does not crash when an invalid song is requested
  • Use Perl formatting (perldoc perlform) to generate output when listing things in the command-line interface; for example, list songs using a format string so lines don't wrap and whatnot
  • Use Term::ReadKey for handling the user input on the administration interface; Term::ReadKey can turn off the echo, which would be great for passwords
    • system("stty -echo");
  • Net::TCP::PtyServer can connect a socket to a PTY device, which is all that a telnetd daemon does
  • Look at using IO::Socket::SSL for encrypting communications between the client and the admin server; most likely a client script to use this method of control would have to be created as well
  • create a 'wrapper' for both client and server sides, which would wrap the stream inside of SSL encryption; the server would serve packets to the wrapper, which would encrypt them and send them out to SSL clients, who would then receive them and make them available to the player via a local TCP socket
  • add support for named arguments (key => "value") to object methods; inside the method, you shift the object type from the left hand side of the argument stack, then the rest of the arguments can be assigned to a local hash
    • my %args; %args = @_;
    • check also for an uneven length of @_, and throw an error if uneven length is found (see the article on using Perl's AUTOLOADer for an example of this)

User Interface

  • use a "psuedo-shell" Perl module to provide the shell mechanisim for Streamcast's console interface. It would provide neat things like command history and tab completion of commands and filenames
    • Term::ShellUI - complex, complete
    • Term::ReadLine::Zoid - complex, complete
    • Term::Shell - simple
    • Term::Complete
    • (Search CPAN for "tab complete")
    • Each module that registers with the main Streamcast object, should also be able to add menu items to the Term::GDBUI menus, so that the menu items become dynamic based on which modules are currently loaded.
  • long output inside of the shell should paginate; allow paging forwards and backwards by line and by page
  • Write a GUI front-end thingy using UI::Dialog
    • Write a UI::Dialog backend module for CocoaDialog, so you can run Streamcast/Propaganda on Macs
  • Add output "themes", which allow you to generate output according to a template for different output mediums
  • Use Locale::Maketext for i18n

Access Control

  • make user management fucntions, especially things that control who can play what when, so that in order for people to get more ability, they need to have it given to them by the admin. Check for duplicate users before commiting the rest of the user info to the database
    • skip song priviledge
    • die/restart priviledge
    • console login privilidge
    • Added access field to users table,
    • level 0 access == superuser
  • Create an access control system so that when a user tries to do something, that table is checked to see if the action is allowed

Logging

  • Wrap/truncate output of streamcast console when it is used interactively, based on the environment setting for terminal width ($COLUMNS). When the console is being used by another script, don't do any wrapping/truncation.
  • record an error in sql_evt_logger when a non-existant song is played
  • allow level of logging to be controlled via a settable variable inside the interface
  • assign different log severity levels to different actions (ala syslog), so important actions/events always are seen, and the unimportant stuff is filtered out
    • a song missing on the filesystem but still in the database would be an ERROR for example

Song Queues

  • allow for combining multiple data sources when creating lists of songs to be played. The only stipulation is that the files need to be readable via the Perl read() function.
  • Use URL style notation for selecting data sources
  • Does StreamCast utilize a song_q ala StreamDb for storing songs that haven't been played yet, or does it pick songs at random from the song hash with the possibility of repeating songs?
  • Create a "Suggested Songs" queue; pick a random list of 5/10/20 songs, and let the user/operator choose amongst them for songs to queue/play

Database Operations

  • see how long all of the different types of metadata can be, and adjust the SQL scripts so that the database fields will be large enough to hold the information
  • FollowPlay mode - tag one song with the song_id of another song to play immediately after the first song. The songs would need tagging, and a setting in the control interface would have to be created so FollowPlay mode could be enabled or disabled
  • have stream-db read the tables in the database on startup, so that you can detect what version of the SQL file is currently being used, and recommend an upgrade if the 'SKIP' field is not present. (add version table to database, have stream-db check on startup)
  • add the MP3 filesize to the database for use by external scripts
  • add .m3u playlists to a new table, normalized to the mp3main table; read the songs in the .m3u playlist, and match them to songs already in the database; call this the albums playlist, meaning it holds complete albums
  • have the ability for users to create their own playlists
  • fix up the mp3tosql.pl script so it walks the directory tree looking for new songs that are not in the database, and songs that are no longer on the filesystem

Support Modules

  • Ratings.pm, creates tables in the the backend storage mechanisim that keeps track of ratings songs for individual users, as well as a summary for all users. Import ratings from iPods too
  • Create a MP3 streaming multiplexor, reads sources from multiple MP3 streams and relays those streams to another server or to clients
  • add ability to play commercials every X minutes. And songs
  • Add graphs on usage for requests/etc only during certain times of the day.

Support Scripts

  • Found a copy of ripit.pl
  • come up with a better MP3 ID3 tagging tool, one that will re-tag all the files in a directory and use the parent directories to determine the album and artist names. Try using MP3::Info from CPAN (Mac-iPod-GNUPod uses it)
  • split file and directory names on spaces and/or underscores, so as to match all of the various filenaming schemes in use
  • add metacharacter replacement in filenames during ripping; keep a list of metacharacters that should be replaced prior to saving the ripped file; for example, replace ()[]/\*$ in filenames with underscores or dashes automagically, but let the user make changes before commiting files
  • create a java application for controlling streamcast and icecast servers
  • Change the ripit.pl script so it does it's own freecd queries and parsing, instead of relying on the user to have xmcd/cda installed and set up correctly. There's quite a few scripts/binaries that will get the CD's magic number which would allow for a freedb lookup
  • Both cdparanoia and cdda2wav will get the CD's DISCID information
    • cdparanoia -Q (needs to calculate DISCID; see links below)
    • cdda2wav -J -v toc
  • How will you handle multiple returns to a DISCID search?
    • retrieve all of the records for each DISCID entry and let the user choose which ones they want to use
  • Add the dburn-wavs.sh script to the support directory, or perl-ify and add it as a modular plugin to Streamcast
  • Vorbis Comment tags
  • See freedb.org for info on how to generate unique ID's for audio discs for looking up on FreeDB.
  • Debian has a makepasswd.pl package that generates crypted passwords; adapt for both crypt and MD5 passwords?
  • mkpasswd.pl script for generating crypt'ed or MD5 passwords for use in streamcast

Packaging

  • Use PAR.pm to create PARchives of all of the packages needed to run on a specific architecture.
  • Create a Perl bundle, so users can load CPAN and call install Bundle::Streamcast to download and install all of the Streamcast scripts.

Testing

Adversity testing should also be made a part of these scripts, so if someone else were to pick them up and try to use them, they would actually work.

For each atomic operation inside of StreamCast, a test should be written for that operation.

Network testing

  • Test out the telnet interface on the Mac; it's a BSD box, so if sockets are broken on BSD, it should break on the Mac
    • write a simple socket tool that can either listen as a server or connect to the server to perform different tests

Command line testing

  • Come up with different command line options that may hang AppConfig, verify that AppConfig does the right thing

Testing filehandling

  • Generate a set of files for testing; each supported format should have it's own file in that format included in the distribution, and the file could be played to verify that decoding that format works
  • Try parsing filenames that contain multiple period characters, spaces, tabs, other non-printable characters (this.is.some.file.name.jpg) to see if the regular expression parser for file suffixes is solid.
  • Try to test/detect problems with read-only files/directories

Testing Webpages

  • run the tests through the HTML/CSS validators, and fix any issues that the validators find

Reencoding

Moved to Streambake Wiki

Unsorted

None. Fixed 13Jan2006

13Apr2003 - found this error in stream-db today: 

 > add req 5591 
 DBD::mysql::db do failed: You have an error in your SQL syntax near '6',NULL)'
 at line 1 at /usr/local/icecast/bin/stream-db.pl line 481, <ADMCLI> line 21.
 You have an error in your SQL syntax near '6', NULL)' at line 1 at 
 /usr/local/icecast/bin/stream-db.pl line 481, <ADMCLI> line 21.
 Connection closed by foreign host.
 [observer][brian brian]$ stream: playlist disconnected, terminating.
 stream: Caught sigint, shutting down

Completed TODO's

27 May 2001

  • IC - stream the song title, album and artist metadata
  • IC - show -> uptime; record when stream-db starts, and do an uptime counter

Previous

  • DB - add the default system users to the .sql file
  • LO - have a table that tracks things like:
    • console access
    • request frequency, play_log can be searched to see when things get requested (event_log and play_log tables have been created, there's really no limit to what you can log)
  • IC - change the checking of command words so that they are regexp matches instead of exact matches
  • LO - have a song_q reload tracker, that tracks when song_q gets reloaded
  • LO - have a song tracker, that tracks who requests/plays what when
  • LO - have an event log that captures events,
  • PL - playlist -> populate
    • copies all of the songs from mp3main to the song_q, so they can be removed as they get played. Only ID's need to be copied over
  • IC - function will check for 0 songs, then repopulate if there are no songs
  • IC - song_q -> delete: delete a song from song_q (when it gets played)
  • IC - stream -> get_filename: pulls the filename out of the database, so it can be passed onto the stream
  • IC - authuser ->
    • Don't need a crypt password file anymore, that's what MySQL is for
    • if the user has an extra char (#) in the username, then don't call mysql passswd when you run the select for that user (this may not be needed, hopefully the front end stuff will just add/delete songs directly to the request_q, and the console reserved only for changing settings in streamcast by admins only)
  • IC - server_command -> get_next_song >DONE<
  • IC - sort of done, a random song will be pulled from the database.
    • no idea if non-random works at all
      • get a random song from song_q -OR-
      • get the next in line non-random song from song_q -OR-
      • get the next song out of request_q

then...

  • remove the song from the queue, inserting it into the play_log
  • IC - stream -> get_next_song: change the get_id3_info from the individual file to pull from the database instead
  • IC - playlist -> __init__ : check song_q to see if songs need to be loaded into it, before everything starts (song_q gets reloaded at program start anyways.)
  • IC - playlist -> add -> request: add the request number to the request_q table
    • use the mp3main DB to pull info about the song back out, and display it for the user
    • combine song display with p->show->info into it's own function
    • log the user's request into event_log 
 playlist -> show -> info >DONE<
  • see playlist -> add -> request
  • pull song data from the database 
 playlist -> show -> requests >DONE<
  • print the contents of the request_q 
 playlist -> show -> current >DONE<
  • pull the last song from play_log?????? 
 playlist -> show -> current -> short
  • show a short version of the current song, 1 line with all the info 
 playlist -> show -> settings
  • remove the perl playlist stuff, and add the database info instead 
 playlist -> reload -> playlist >DONE<
  • reload the song_q
  • see playlist -> populate 
 playlist -> set -> playlist
  • set this to a different database??? This will probably get removed. See below 
 playlist -> search >CANCELLED<
  • add another parameter that sets up what gets searched for, songs, albums, filenames, artists (decided to search in filename and filedir, keep it like the regular streamcast, if you want to search off of other fields, write the functionality into your front end scripts)
  • search from the song database, and create a normalized album database, so you can play whole albums at a time.

Functions that will get removed

set -> playlist (all of set will go). Possibly make this into a database changing function, i.e. you issue this command, and you can change which database is getting loaded into the song_q. Maybe not...

playlist -> show -> songs: show 3300 songs? get real

Misc info

From the MP3:Info perldoc: 

       get_mp3info (FILE)
           Returns hash reference containing file information for
           MP3 file.  This data cannot be changed.  Returned data
           includes MP3 version (VERSION), total time in minutes
           (MM) and seconds (SS), boolean for STEREO, MPEG layer
           (LAYER), BITRATE, MODE, boolean for COPYRIGHT, and
           FREQUENCY.

       set_mp3tag (FILE, $HASHREF)
           Adds/changes tag information in an MP3 audio file.
           Will clobber any existing information in file.

           Fields are TITLE, ARTIST, ALBUM, YEAR, COMMENT, GENRE.
           All fields have a 30-byte limit, except for YEAR,
           which has a four-byte limit, and GENRE, which is one
           byte in the file.  The GENRE passed in the function is
           a case-insensitive text string representing a genre
           found in @mp3_genres.

- genre for Movie Trailers is 70

Back to the top of the page

view · edit · attach · print · history
Page last modified on April 04, 2010, at 11:04 PM