STAR Election Results

[mcg23]

Sorry for the technical difficulties, STAR has been down due to its popularity tonight. Follow us on http://www.twitter.com/standrewsradio for live election results!

-STAR

[Delts]

I think my feelings on the coverage was made well known last night on twitter, but for star to go down when 80 people listen in is a disgrace. Even when the coverage was restored the volume of the stream was absurdly low and the static on the left (iirc) channel was unbearable. And yes, the twitter coverage was good, but it's St Andrews Radio, not St Andrews Twitter.

[AliceChalliner]

Give them a break! you know what its like when things fuck up on the ents crew...sometimes it just doesnt work, im sure they were doing their best.
Plus it's not like the union will buy them equipement to meet that high a demand when they only get that many once a year. A lesson learned for next year to hire in some bigger equipement for the one night/week id say....

[Fawksie]

I don't think hiring servers physically is an option offered by many (any?) companies, and it's not really feasible even if it was. The audio hardware is all now up to a commendable standard, but the studio servers are second-hand desktop PCs which are in need of replacement. No doubt Walker, being current station engineer and IT chief, is acutely aware of this and will see it done during his term as Broadcasting Officer.

[Andrew W]

I feel I should clear a few things up here, and thank some people who really should be thanked.

Firstly the old PCs in the studio are no more, we upgraded them a few weeks ago to quad-core chips and other things I've never understood. But I'm told we're running a hugely powerful system which has a degree of future-proofing. The audio hardware is truly excellent and now it has been set-up how we need it works perfectly.

Friday night was extremely disappointing. Firstly I want to thank the STAR News Team, especially their Head of Department Cordula, because they provided first-rate interviews and reports quickly and well and it's just a shame we couldn't broadcast more of them live. In addition the efforts of Walker Angell, your new SSC Broadcasting Officer, who worked tirelessly to try to bring the feed back should be recognised. The low quality efforts were part of this as we tried to bypass the site of the problem however it was not possible in the end.

I'm aware that Delts, and one other Twitter user who was particularly insulting, are cynical about all things STAR and thus I don't read a huge amount into what they say about the station. I was one of the four presenters on air for the show so can tell you how it really was. After a couple of system reboots to ensure the problem was not at our end our playout software and systems were working perfectly, indeed we presented the remainder of the show. Unfortunately nobody could hear it. Again, I've never taken much interest in IT so correct me if this doesn't make sense, but I believe our website and stream are hosted on University servers on the North Haugh. When they have power outages down there we have been known to have problems. I believe the problem lies with this part of the output. We tried to fix things from the studio but in the absence of 24 hour IT support at the University there was little we could do.

I would like to thank the ex-STAR committee members who so kindly offered to donate their expertise and server space when we were trying to get online again. We declined these offers to prevent 100+ users rushing over to their sites and potentially causing them a similar problem to ours. Their actions restored my faith in our radio station and made me realise what a valuable asset it can and will be to the student community in St Andrews and beyond. The petty and uninformed insults and criticism that were directed towards us on Twitter on Friday by a few users doesn't detract from this.

80+ listeners is excellent for a student radio. Few people appreciate that STAR outperforms student radio stations at much larger institutions and is something we should be proud of. In the next academic year we will endeavour to prove our worth to you and continue the excellent job started this year by Phil Tortoroli and by Walker.

Andrew Watson
Head of Production 09-10
St Andrews Radio

[steelegbr]

Unless things have epically changed, you currently run all your services on starfm.st-andrews.ac.uk including file server, authentication, web site (Drupal talking to a DB backend), PostgreSQL (website, old playout), MySQL (Rivendell) and so forth. I don't know if you still read all audio off the network. I'd hope not with Rivendell, punting raw WAVE at low latency over the network can't be fun.

My understanding is there was a pretty epic failure on the server side of things during this event. Basically we're talking I/O issues, kernel panic, the works. There's also a rumour a well timed cron job was the straw that broke the camel's back. So it was game over for anything running on the server, which in STAR's case is everything.

Yes, you don't have 24/7/365 IT support to fix it so any problem out of hours can't be dealt with too well unless you're lucky and someone at ITS notices. I'd be very surprised if your services were high priority where multiple failures to occur with the uni computer systems.

As for the listener numbers - well done. However, your kit should be able to cope with far more than that no problem. It's not unknown for servers of a similar power and network connection to handle hundreds of listeners. Though, that problem could be down to the whole run everything on one server approach that you can't feasibly get away from.

Besides, STAR's IT systems are not the absolute worst I've seen in the industry (though certainly nowhere near the best). The entire C drive of the computer doing the audio logging for for Ofcom shared to the network anyone? And no, before you ask, I'm not naming the station in question.

As for the audio processing - I'll leave my opinions on that one out of here. Let's just say that for live presenting, I'm a fan of heavily processing microphones. Oh and has use of the PFL buttons on the desk expanded at all recently? (I would really hope so - they were used as ornaments by most people during my time there).

[donpablo]

I didn't hear the broadcast so I cannot comment on the night itself but from my brief experience of STAR it is a bit of a shambles. Why STAR even needed so many months off to fix things that clearly still aren't fixed is beyond me. It has what? 300 members? And they can only manage 80 listeners for a big event of this kind. Pretty poor when you know its all just your mates and maybe mom and pop. But yep, let's all pat each other on the back and give ourselves a big high five on a job well done.

[Delts]

To state that I'm cynical and then just brush off my comments like that is rather poor in my opinion. Whilst I currently think Star does a very poor job and is a waste of money considering the average number of listeners, it has great potential.

The fact however remains that 80 people listening should not cause serious difficulty (and whilst I understand there was problems at the university end, being dependant on their system which goes down rather too frequently is certainly far from ideal), when the feed was restored the gain was far too low (comparable to other things on my computer the volume difference was absurd) and the static was unbearable. I fully understand that when things go wrong it can be a major pain, as was experienced whilst I was an Ent as Alice points out, but there was just too many failings overall.

Just looking at your twitter feed, "The University Server that runs our stream and website has crashed because of too many hits". I do believe that this isn't correct from what I've been told by others and if so then why say it, but Star should be more than capable to deal with the server load. 80 people is not a lot in the grand scale of things for online radio.

[Fawksie]

Firstly the old PCs in the studio are no more, we upgraded them a few weeks ago to quad-core chips and other things I've never understood.

Colour me informed, I thought that you'd managed to get Kieran to fix starfm (the North Haugh server) and that the studio encoding server had then died on an unrelated note, which is why I suggested on Twitter that you take a look at Darkice.

WiredSoc's servers have suffered the same type of failure as what happened to starfm on the night, and it's not necessarily linked to high load. As steelegbr mentions, it would be useful to pull as many services onto the studio servers as ITS will permit, in order to decrease both the load on the network connection and the dependency on ITS intervention to rectify failures.

[Fawksie]

Just looking at your twitter feed, "The University Server that runs our stream and website has crashed because of too many hits". I do believe that this isn't correct from what I've been told by others and if so then why say it, but Star should be more than capable to deal with the server load.

It appears that the university server was crippled for most of the time that we were listening, and wasn't fixed until Saturday afternoon (by Kieran no doubt doing unpaid overtime). I assumed it had been fixed during the broadcast, and I attributed the audio distortion, probably falsely, to problems with the studio server.

[tv32]

I assumed it had been fixed during the broadcast, and I attributed the audio distortion, probably falsely, to problems with the studio server.

Yeah, nothing was fixed during the broadcast. The postgreSQL database, hosted by the University, hit a limit for maximum number of non-superuser connections. This is most likely due to the fact that postgres databases default to 100 maximum concurrent connections. When the limit was reached, the postgres database appears to have crashed instead of simply turning away new connections which lead to a breakdown of the stream as well, and attempts to restart the stream didn't work. The stream was, for a few minutes, switched over to a someone else's icecast server to verify that none of the issues were on our side. Running on a non-University housed icecast server, STAR ran and sounded great, but we didn't have permission to publicize the server that was loaned to us. None of the studio equipment malfunctioned, darkice was not the issue.

While I agree with Delts that the stream crashing at 80 listeners is shameful (and it was 100, not 80 by the way), it was not a fault of star. As far as I'm aware, the 100 limit that is default coding in postgres was never changed by the University, and since we never hit it before, it was never discovered as a potential problem. But the twitter feed was entirely accurate. It was the university server crashing due to too many hits that caused star to go offline.

[Fawksie]

The postgreSQL database, hosted by the University, hit a limit for maximum number of non-superuser connections. This is most likely due to the fact that postgres databases default to 100 maximum concurrent
connections.

Correct.

When the limit was reached, the postgres database appears to have crashed instead of simply turning away new connections which lead to a breakdown of the stream as well, and attempts to restart the stream didn't work.

Incorrect. Such a failure would be a truly massive bug in PostgreSQL. In any case, the only service dependent on PostgreSQL is the website, and failure of PostgreSQL would not have affected Icecast. What actually caused the server to be partially crippled was the failure of part of the Linux kernel, most likely a module dealing with disk IO, hence the drastic slowdown of all services and audio corruption on the stream.

It was the university server crashing due to too many hits that caused star to go offline.

As I said in a previous post, I don't believe the fault was load-related. In any case, ITS should have been contacted. The IT team ought to know by now who is responsible for the server and even if unable to contact him directly, the ITS out-of-hours line would have been able to put you in touch with him.

[steelegbr]

Running on a non-University housed icecast server, STAR ran and sounded great, but we didn't have permission to publicize the server that was loaned to us.

Intriguing. Kinda kills the point of running the server beyond doing some simple diagnostics.

Fawksie was also right (as far as I understand) about the cause of the failure. I'd assume that's becuase we use the same source.

However, the DB server failing should just display a generic Drupal error message on the homepage. Sort of some pretty epic resource usage from PostgreSQL, it should have no real effect.

As for Delts point about listener figures - well, local radio stations operating with a sizable level of local content tend to see little more than about 30-50 listeners on the web stream maximum. Most listeners come from FM broadcast and can be in the thousands. These also tend to operate on a mass appeal basis music and content wise.

STAR can at time be extremely specialist listening, even during "peak" hours. Unless STAR start to go more mainstream, you can't expect listener figures at the level they got for elections unless there is a special event on. Now whether that is value for money when it comes to the union is a long debate. We could also drag out the LPAM options in that one.

Also, very few radio stations operate the web stream servers themselves. The service is often purchased from a provider at a fixed monthly cost and with a fixed listener limit.

There's also the whole training issue that seemed to drift in and out of vogue at STAR when I was there. The approach to training was simply a half-hour session in the studio then "away you go". You really ought to have experienced personnel with good habits (level checking / use of PFL, microphone usage, running to time, etc.) taking apprentices on board until they are competent enough to go it alone. Heck, there are hospital radio stations just down the road that take this approach and will not let you go to air until the "trainer" is happy and your demo based on this training is accepted by the committee.

[tv32]

Incorrect. Such a failure would be a truly massive bug in PostgreSQL. In any case, the only service dependent on PostgreSQL is the website, and failure of PostgreSQL would not have affected Icecast. As I said in a previous post, I don't believe the fault was load-related. In any case, ITS should have been contacted.

Yeah, blaming it on the postgres database was probably wrong on my part, as it would be strange to cause the other problems. I'm a bit over my head in the technical stuff... thankfully I don't have anything to do with it at star. Despite that, I'm going to take one more stab at it. As for it not being load related, you could be right. However, the error message that drupal returned if you let the standrewsradio.com website load for about five minutes was that the maximum number of connections was reached, which suggested that it was indeed load related. Perhaps this error message was wrong, or perhaps it was a huge coincidence that we reached the maximum number of connections at the same time as the server failure. Perhaps more likely it was simply a perfect storm of conditions that all lined up at the same time.

As for contacting ITS, by the time star had completely figured out where the problem was, it didn't make sense to call. When the website and stream crashed, our systems were rebooted and the stream was re-established, albeit with poor quality. The focus at that time was fixing the stream, not the website, and there was a conflict with jack audio and darkice that seemed to be causing the problem at our end. It was only after several reboots with different configurations and other attempts to subvert the problems that it was found through some frantic web searching that the errors the system was having are found generally when the system has connection problems with the server. As I said, at that point the stream was switched to a different icecast server not university housed to make sure that indeed the problem was not in our systems. With that server, darkice and jack ran great and the audio came through fine, with no static, and with the levels fine. By the time we were sure that the stream problems were caused by the same issues as the website, it was incredibly late at night, and didn't make sense to wake people up (both numbers the union had are people with young children) especially since we had been down for several hours by that point.

Intriguing. Kinda kills the point of running the server beyond doing some simple diagnostics.

The point of the server being loaned to us was indeed simple diagnostics.

However, the DB server failing should just display a generic Drupal error message on the homepage.

Eventually, after a while, the connection limit message would be loaded by drupal

There's also the whole training issue that seemed to drift in and out of vogue at STAR when I was there. The approach to training was simply a half-hour session in the studio then "away you go".

A big part of restart was indeed to get decent training in. For the two weeks back after break when star was still offair, all on air teams were in the studio, being retrained, running through rehearsal shows, and learning the new software. That included learning how to pfl and adjust levels.

[Delts]

As far as the leveling goes, I tuned in again last night and still found the gain to be too low. I'll take a look later at exactly how much lower it is compared to other audio systems that I use, but it's a case of the overall output volume just needs kicking up a bit.

[James Shield]

Quick question to those of you who know your way around the STAR website: would it be possible for the RSS feed on the Podcasts page (http://www.standrewsradio.com/audio/feed) to list all of the items in that section, rather than just the ten most recent ones?

If this could be changed, it would allow complete access to STAR's back catalogue of podcasts via the newly created iTunes page:
http://bit.ly/starpodcasts

[DACrowe]

Quick question to those of you who know your way around the STAR website: would it be possible for the RSS feed on the Podcasts page (http://www.standrewsradio.com/audio/feed) to list all of the items in that section, rather than just the ten most recent ones?

If this could be changed, it would allow complete access to STAR's back catalogue of podcasts via the newly created iTunes page:
http://bit.ly/starpodcasts

A history podcast I listen to ran into the same problem recently; they were forced to change the host they were using (from Blogger to WordPress IIRC) so it probably depends on the RSS software being used, but I don't know enough to say for sure.

[steelegbr]

...so it probably depends on the RSS software being used, but I don't know enough to say for sure.

The system STAR makes use of is something called Drupal. There are plugins available for integrating with iTunes. But even if the support is not there to get the entire history in the RSS feed, I'm sure they can poke around in the source code to make it work. Shouldn't be too difficult to find the right module and poke it.