Icecast -kh branches

This is the current set of branches of ices2 and icecast2. They extend on the offical release (currently 2.3.2) with features that may be (if found to be working out well) merged into the next offical release. You can contact me karl <at> xiph.org

Main releases

• kh20a. crash fix in cases with a relay removed/disabled
• fix possible lock imbalance bug if mutliple workers set
• make sure listeners are not paused if relay is not running
• fix memory leak on each source disconnection.
• fix double free corruption bug if listener_remove used.
• on server shutdown, prevent auth of new listeners only. other auth requests can continue which allows client details to clean up properly
• relocate where the source count is increased. For source clients it is just before where it used to be but for relays it is increased before a relay connection is attempted. Bug in kh19 meant sources kept increasing on relay restart
• fix possible stalling bug in auth command method.
• listener_connections stat could have a random data bug
• Do not force a master relay recheck on stream start/stop
• build fix on certain platforms.
• small internal code cleanups.

• add 7.xsl processing to present stats for a shoutcast /7.html request, requires alias of /7.html to /7.xsl, not complete but basics are there
• avl tree of listeners is back in, workers process lists of clients but the tree enables a quick listener lookup from a source.
• the admin arg require handler left a lock held if not specified.
• make sure the allow/deny/useragent files reload properly
• initialisation bug for auth, more likely to affect win32 gui.
• content-type has to be lower case for shoutcast to recognise the mime type.
• per-mount access log. <accesslog> <name>/file.log</name> ....
• update to the source shutdown process. reduces code in relay shutdown, no point in duplication. allow listeners to be placed in a waiting state if source is waiting to start up.
• do not dump stream stats on on-demand relay failure.
• allow for reporting xml parser warnings to stderr at startup
• fix possible auth related corruption with disconnected clients on slow auth
• type fixup on lag calculation on 32bit systems.
• if streamlist fails then drop all slave relays.

• for people who use ICE_MUTEX_ABORT (environment var) to trap for long held locks, change the name to ICE_LOCK_ABORT
• fix possible memory corruption case in stats and threading
• fix vorbis stream bandwidth increase when certain source clients connect
• YP thread now started from worker as required instead of once per second
• avoid any ::ffff: in IPs
• shutdown order and other internal code cleanups
• use signalfd if available for processing signals in connection thread, saves waking up the thread.
• make sure the auth mountpoint returned is used instead of the listener supplied one. Allows for redirecting listeners to alternative mounts
• update command auth, works in a similar way to url auth now.
• stats clients work better under much heavier load. stats notifications are grouped together if possible before sending, this reduces writes in the many streams case

• fix possible memory corruption when using multiple workers
• fix possible race case in multiple file handle close
• a few minor timing changes, nothing major.
• kh16b. fix possible crash case with relay failing
• kh16a. fixed a crash bug triggered by flash clients

• you can now fallback to file if the initial mountpont is not available as long as there is a limit-rate set at some point within the fallback chain.
• A header from url listener add "icecast-auth-user: withintro" indicates that the response contains intro data to be sent to the this listener. Anything else indicates the previous behaviour.
• allow source clients to be limited by duration, set by time limit header in url stream auth
• internal changes to average bitrate calculations. blocks are now removed based on the timestamp not the block count.
• fix possible race with url auth'd listener getting a 404 response
• try throttle file sending based on duration/bytes sent on file handle and split the file sending function into two (throttled/unthrottled). reduces memory.
• merge in compile-time allocation counters
• Added some stats for fallback files, and metadata_updated for sources
• minor cleanups for internal API. log message cleanups.

• lock mismatch identified, could cause a stall in the worker.
• relay connection timeout was not being set in all cases
• Do not fallback listeners on relay fail if server shutting down
• minor code cleanups.
• 15a. fix memory corruption when many clients are referring to the same file.
• 15b. fix possible crash on url listener failing.

• Allow stats/listclients/killclient work with file references
• Add timeout parameter to relay. number of seconds for connect to complete.
• fixes for restarting of relays, especially where changes are detected and where fallback to files were set.
• repeat fallback file
• tuneup client scheduling for file sending
• added a few validation checks, useragent chars, xml config port ranges
• log message cleanups.

• on source exit, listeners do not need to be moved to the file serving engine, just release any shared data and delete client if not on worker. saves process work at source exit
• fix possible crash bug with long lock held detection.
• allow listener_add response to include intro content
• merge connection_t into client_t, and shrunk connection_t, reduces memory used
• improve some of the internal client scheduling.
• the reading of the mime types file was done too early
• admin/fallback now updates the mount details if available.
• more source lock cleanups. nothing major
• change the order of the header routines, the format specific routine now calls the general routine.

• xml based relays could cause a crash on server exit or xml reload
• avoid memory/fd leak on active relay restart.
• cleaner worker thread shutdown.
• win32 update. Prevent Esc closing the GUI. Make the service work better and is now able to take an alternative xml as parameter. Update libogg in the build to 1.1.4 and refer to the current directory for dependent DLLs.

• busy looping caused by truncation on 32bit setups.
• avoid possible deadlock between file release and open.
• relay client handles were not being cleaned up fully.
• send Host: header on relay connects.

• big internal thread change. instead of one thread per stream, we now have a fixed number of worker threads, each processing a set of clients (sources/relays/listeners etc).
  • no source or fserve threads now. auth still has thread(s) as the API for libcurl is blocking
  • worker threads with no clients sleep for long durations.
  • defaults to 1 worker thread, overridden by <workers> in <limits>
  • listeners get moved to the worker thread running the source, helps caching
  • sources get moved to a less busy thread if it would help
  • each client has a timestamp to indicate when it wants service. The sleep duration for the worker is based on the next client to be serviced.
• if fallback to file is specified and the requested mount is not active then a 404 response is returned instead of the file contents.
• fserve engine can handle bandwidth limiting per client. Fallback to file uses this by using the average incoming bitrate of the source stream as the target limit.
• file serving uses an internal cache of open files, so that 1000 listerners do not suddenly open 1000 files if a fallback to file occurs.
• move listeners admin request can specify another source or file.
• relay changes
  • the client structure is always allocated now, instead of when the relay is started.
  • listeners stay on relay if switching to another relay master. Only when they all have failed does the fallback apply.
• post-kh9 fixes
  • fix for possible race with rejected new listeners going through auth
  • make mp3-metadata-interval 0 work again
  • auth htpasswd checks for details provided like user:pass@host:port/mount and if not then checks as host:port/mount?user=a&pass=b

• allow shoutcast source client auth work via stream_auth url
• allow for a flash policy file.
• internal cleanup to help merging into trunk
• fix vorbis metadata update via url, broken in recent releases. cleanup some metadata updating/logging
• drop vorbis stream rebuilding back to flush evert half second, in sync with trunk
• stop false reports of 'not null' in debug logging. no other effect on server
• limit how often the relay mountpoint "already exists" logs.
• minor tweaking to the average bitrate calculation
• remove the internal global time caching, although prevent several time calls if in succession.
• kh9a fixes a race case which could cause a crash with rejected auth'd listeners

• fix memory corruption which affected ogg streams.
• fix for unsupported FBSD/win32 spin locking options. abort on initialisation failure.
• win32 fix from kh7 causing stalling source threads. fallout from kh7 build failure

• merge fix for bug report on trunk for skeleton
• fix for build failure with recent glibc
• fix for admin.cgi metdata updates not applying (kh5 bug).
• minor locking cleanup
• win32 fixes. server restart could cause relay details to become corrupted
• win32 update. updated libxml2/libxslt

• some source stats were not being reported on the xsl pages.
• win32 access log timestamps should now to correct.

• stats fixes, crash case in stats handling, and hidden attribute was not honoured on xsl pages with recent stats changes.
• if requesting /admin/streams via stats method then allow auth via relay user/pass or mountpoint auth. receives only stats for slaves
• allow url= tag on metadata update for non-ogg streams, adds StreamUrl.
• fixed possible crash with skeleton stream handling
• actually prevent metadata updates if there is a stream already running from a different IP.
• fix build bug on FBSD7
• allow optional queue-len tag in listen-socket, default is 10

• Where's the brown paper bag. boolean settings in the xml were being ignored due to an incorrect test. update from kh3

• fixed master not informing slaves of streams bug. caused by missed setting in the kh2 stats update
• more xml checks for badly formed xml files.
• log files can now be in sections not just a single filename ie <errorlog> can contain <name> <archive> <display> and <level>, accesslog can contain <name> <ip> and <archive>, playlistlog can contain <name> <archive> <display>. In each case, name is a filename, archive is a true/false setting for using timestamps on a reopened file, display is the number of lines to show in the admin page.

• more stats work. Clients are now processed as needed instead of a dedicated thread per stats client.
• do not allow raw metadata updates if not from the same IP address as the source client unless it is from the admin user. For some reason some source clients issue updates even though they are rejected.
• missing lock on streamlist master/slave update, could cause memory corruption.
• update to average bitrate handling. feeback please
• allow strftime expansion on dump filename, applies at the time the stream has started
• removed debian directory, it's not maintained really so leave it to the debian people to deal with it themselves.
• experimental theora keyframe as png patch added. not built by default
• minor memory leak plugged in fserve

• remove stats thread. Stats clients still have their own thread and queue.
• fix low bandwidth theora stream problem.
• failing on-demand relays were not being recovered as they should of.
• encode xml entity data for clients useragent/username
• reduce memory usage in avl nodes, seems to benefit win32 more because of the unused lock handling
• minor patch for kate detection applied. ogg.k.ogg.k
• minor type cleanup for off_t in fserve

• Added Kate/Skeleton codec handling within Ogg streams. patch by ogg.k.ogg.k
• small changes for stream directory handling, mostly for error cases.
• added /admin/reset?mount=/mountpoint&setting=x to reset stats for that mountpoint. without setting= all resetable stats are reset. x can specify peak,read or send or a combination of them (comma deliminated)
• /admin/streamlist.txt uses the same routine as /admin/streams so does not have a limit on how much is sent back.
• updated win32 code, windows service should be ok again. service name now includes version to help multiple installs but windows may require a reboot between removing a service and installing that same service again.
• fix possible crash on reload of xml with bad tags
• small update of web pages
• added per-mount stat for total mbytes sent
• kh34a, avoid possible loop on yp add failure case

• master/slave update. slave mode only issues a streamlist.txt request if the /admin/streams request fails. /admin/streams also acts as an mountpoint for the slave relays by passing a mount= arg and auth. You can now define a <mount> for /admin/streams to specify how slave authentication is done eg url (via listener_add) or htpasswd. Once authenticated, the slave can bypass any limits like max listeners etc.
• The slave also allows for defining a <master> tag block instead of <master-*> tags allowed are <username> <password> <server> <port> <redirect> <bind> auth url can take an "icecast-slave: 1" header, which makes client act as a slave and bypass mount limits.
• build fixups for OpenBSD and cases where IPV6_V6ONLY is missing
• move 'clients' limit check to after admin request check, means admin commands can work even if the limit is reached. removes a lock taken as well.
• drop connections to known slave relays if we find that the destination source is a fallback to file, as this would cause high bandwidth usage.
• stats fix for inactive on-demand relay stats without a mount definition.
• added some minor consistency checks.

• crash fix for a failing on-demand relay.
• incorrect logfile settings for timestamp setting, for playlist and access.log
• update for win32 build and initialization.

• hard to trigger but possible race condition on source startup.
• long standing bug identified with many descriptors (eg > 1024 listeners)
• allow YP thread to skip a non-responsive server

• crash fix if no user/pass supplied when stream_auth in use.
• the admin=1 was not passed in POST for stream_auth on metadata updates

• some small code re-organisation in kh28 caused auth htpasswd to fail new listeners
• work around FBSD issue with listening on IPv4/IPv6 sockets

• race fix in yp thread startup.
• manageauth userlist page was blank due to a NULL check, fixed.
• add rejected_mount auth option to redirect unathenticated listeners to an alternative mountpoint. eg subscription stream redirected to low bitrate
• Allow for url auth to handle 'Mountpoint: ' header for redirecting listeners if they are to be rejected to an alternative mountpoint on the same server. This is as opposed to the Location: header which gets a player to redirect, maybe to an alternative server.

• fix potential fserve thread race
• merge listener inline shoutcast metadata into previous block write. It is very common and saves writing a very short packet per listener at metadata.
• reduce log memory usage when caching lines for web display
• reduce memory usage slightly on 64bit systems, in refbuf/avl storage
• be more 32/64bit clean
• fix possible corruption if reducing qblock size over reload
• get stats thread to update global outgoing_kbitrate stat, no point flooding the stats queue.

• fixed accidental bug which caused big problems for non-ogg streams without a mount defined. With a mount defined the server worked fine.
• if listener_remove used then remove queue reference before auth performed as queue will expand. race case between auth and source thread on refcounting.
• Add error reporting with above case, makes identification easier.
• internal code cleanups.

• kh24 had a lockup bug on reload if auth url used.
• Don't pass listeners to the authentication queue if no auth handler defined
• auth listener_remove was not triggering if listener_add was not specified.
• make sure avl tree for mime types is cleared.
• remove pthread_exit in fserve, older glibc had a memory leak
• fix memory leak/corruption in YP handler, also make YP thread startup instead of polling frequently.
• redo server connection lookup. Now we refcount the server connection details from the xml, make the client refer to it. reduces locking as well
• always call the xml reload from the slave thread.
• fix memory leak over reload, when using multiple master within relay
• incorrect output appended to streamlist output for relays.
• race between stream shutdown and listener_remove could cause memory corruption with queue buffers. Make source clear less strict.
• cleanup any fserve clients after other threads are down, removes lock race
• reduce burst limiter for listeners on sources.
• if available, log file descriptor limit allowed by kernel.

Main differences from offical icecast release

• Auth via url updates.
  • Adds stream-auth option for passing off authentication of source clients to something external.
  • handlers option added to allow multiple auth threads (default 1 per auth).
  • listeners don't keep a hold on the auth_t after auth has finished.
  • Mountpoint: header on reply will redirect failed listeners
• Add reject_mount auth option. sends listeners to alternative mount if failed
• last N lines of log files can be viewed via admin page
• bytes transferred stats for whole server and per-mount
• average bitrate for each stream calculated over last N seconds (default 60).
• option to automatically drop source if bitrate continually exceeds a specified bitrate.
• config xml reload/log cycling can be done via the admin page.
• can specify multiple masters in relay for switching to an alternative stream on failure.
• wildcards allowed in mount-name and IP matching in allow/deny files.
• <filter-theora> mount setting for easy filtering of video from a ogg theora/vorbis stream.
• better handling of xml settings, you can use true/false yes/no for boolean settings.
• single list traversal (instead of AVL) for listeners
• Can trigger a http 302 (redirect) for new listener when on max-listeners reached, can be used to redirect to other registered relays. Registered relays pass extra details to master. work is in the early stages.

Ices 2.x -kh branch

This is the current set of work for a modified ices2. They extend on the official release 2.0.1 release but have a slightly different xml format (see the examples). This includes JACK input support, save file splitting and multiple input switchover.