rtmpdump for Wowza streams

Many sites stream video over RTMP. They hope this BS will save them from people downloading their videos. However RTMP only make their and our lives harder.
So they embed a flash player on the page and provide it with Url to RTMP streaming server like Wowza. All this then lags and is hard to debug. And is supposed to be hard for people to download videos, but fear not.

Today I wanted to download a video from one of Ukrainian news agencies, first look at page source provided me with the RTMP stream URL, well they could hid it but there are other 100% working ways to get it, and there is nothing that can be done on server side to prevent it.

So the URL I got is
"rtmp://91.230.92.3:80/vod/ictv/2011/11/09/mp4:20111109122602.mp4"

To play such stream you can take 'rtmpdump' program which is also available in Ubuntu repositories. Problems start when you try download it and basic parser fails to parse stream name form such URL. Well, normal URL is [protocol]://[host]:[port]/[path], guys who designed all these streaming servers decided to have 2 parts in path [app]/[stream] also there is no way to say what is app and what is stream just looking on the RTMP URL. Most players just have something hardcoded inside or just try all combinations before they find the one that works. In my case, 'rtmpdump' parser thought that "vod/ictv" is the app part and with

rtmpdump -r "rtmp://91.230.92.3:80/vod/ictv/2011/11/09/mp4:20111109122602.mp4" -o out2.flv

I only got - NetStream.Play.StreamNotFound
Usual schema is
[protocol]://[host]:[port]/<one word for app>/<Stream id is everything else>

So to download my video I had to go like

rtmpdump -r "rtmp://91.230.92.3:80/vod/ictv/2011/11/09/mp4:20111109122602.mp4" \

--app "vod" -y "mp4:ictv/2011/11/09/20111109122602.mp4" -o out2.flv

Now it went just ok. Thing may be different in your case, just play around URL.

s3cmd : Delete all files in a bucket

Unfortunately, s3cmd cannot delete files with a wildcard.  You cannot just

s3cmd del s3://my-bucket/*

Well, you can, s3cmd will also report "Object * deleted" but all your files stay in the bucket!
I know why guys in Amazon decided to do so but that is a long story, let's go back to practice.
I suggest a unix-way here (well, I would suggest it anyway):

s3cmd ls s3://my-bucket/  | awk '{print $4}' | xargs s3cmd del

That is all, You will get list of files in an output.

Another way, is to "s3cmd sync" your bucket against an empty directory

Emacs is so Emacs

Today I ran into trouble installing brand new Emacs24(it is still not there, beta or alfa or whatever), I used to have ubuntu this days and ppa is already up at https://launchpad.net/~cassou/+archive/emacs But what is new in Emacs24? Right they finally put ELPA inside Emacs and now bundling it together. The problem is that the default repository (elpa.gnu.org) is somewhat conservative, a bit. Also some packages already bundled with emacs like ruby-mode and this creates more trouble then has benefits.


Now to get some real fun from using Emcas in a Ruby on Rails project you would normally want many other modes and packages like 'rinary' and others but these packages are just not there, not in elpa.gnu.org. To get most of those nice things you have to add some more repositories, here is my ~/.emacs.d/init.el :

(require 'package)
(add-to-list 'package-archives
             '("marmalade" . "http://marmalade-repo.org/packages/") t)
(add-to-list 'package-archives
             '("tromey" . "http://tromey.com/elpa/") t)
(package-initialize) 

Don't forget to M-x package-refresh-contents after you do the init.el. Most packages like rinary reside on marmalade, but current rinary version wants ruby-mode 1.1 and default ruby-mode bundled with Emacs is 1.0 (I guess that is also a version in elpa.gnu.org). And right here we need tromey repo that has new ruby-mode.

Just do it!

So I ran into a "cp: cannot overwrite non-directory" problem. It means I've tried to overwrite a symbolic link with a directory with the same name.
Imagine you have a folder structure:

/folder2/
          /test1.txt
/folder1/
          / folder2->../folder2/ /
                                  /test1.txt


And
/folder3/
          /folder2/
                    /test1.txt
       

To try this out:

:/tmp$ mkdir folder2
:/tmp$ touch folder2/test1.txt
:/tmp$ mkdir folder1
:/tmp$ cd folder1
:/tmp/folder1$ ln -s ../folder2 .
:/tmp/folder1$ cd ../
:/tmp$ mkdir folder3
:/tmp$ cd folder3
:/tmp/folder3$ mkdir folder2
:/tmp/folder3$ touch folder2/test1.txt
:/tmp$ cp -rf folder3/folder2 folder1/
cp: cannot overwrite non-directory `folder1/folder2' with directory `folder3/folder2'

I tried googling but my searching skills were bad today and 'man' page wasn't helpful too. It seems basic cp cannot do the trick.
Now, I do know python/ruby/perl/shell/many other fancy words however I need to copy really large number of files, millions. Estimations show that just plain cp would take a few days to complete, what can we expect from all those less then perfect scripts?

But as one guy (don't really remember his name) said: "Those that can, do. Those that can't, complain" so the real solution is in the very soul of open source:

cd /tmp
mkdir coreutils
sudo apt-get build-dep coreutils
apt-get source coreutils
emacs coreutils-8.5/src/copy.c

Now the diff is:

--- coreutils-8.5/src/copy.c 2010-04-20 22:52:04.000000000 +0300
+++ /tmp/new/copy.c 2011-10-24 17:50:49.633017303 +0300
@@ -1412,10 +1412,21 @@
                     }
                   else
                     {
-                      error (0, 0,
-                       _("cannot overwrite non-directory %s with directory %s"),
-                             quote_n (0, dst_name), quote_n (1, src_name));
-                      return false;
+                      if (unlink (dst_name) != 0)
+                        {
+                          error (0, errno, _("cannot remove %s"), quote (dst_name));
+                        }
+                      if (x->verbose)
+                        printf (_("removed %s\n"), quote (dst_name));
+
+                      /* Tell caller that the destination file was unlinked.  */
+                      new_dst = true;
+                    
+
+                      /* error (0, 0, */
+                      /*  _("cannot overwrite non-directory %s with directory %s"), */
+                      /*        quote_n (0, dst_name), quote_n (1, src_name)); */
+                      /* return false; */
                     }
                 }
 

And build it:

./configure --prefix=/home/sam/new-coreutils
make
make install
cd /tmp
~/new-coreutils/bin/cp -rf folder3/folder2 folder1/
 

Now it works as I need here.

handlersocket problem in Ubuntu Lucid

If you ever happen to see:

mysql> install plugin handlersocket soname 'handlersocket.so';
ERROR 1126 (HY000): Can't open shared library '/usr/lib/mysql/plugin/handlersocket.so' (errno: 2 failed to map segment from shared object: Permission denied)

Just check your /var/log/syslog for messags  like:

operation="file_mmap" pid=12750 parent=12354 profile="/usr/sbin/mysqld" requested_mask="::m" denied_mask="::m" fsuid=105 ouid=0 name="/usr/lib/mysql/plugin/handlersocket.so.0.0.0"

That is a problem with apparmor configuration, the good news - you don't need to stop apparmor to run handlersocket in Ubuntu, you only need to add permissions to your /etc/apparmor.d/abstractions/mysql

Now my config  looks like:

#some comments
   /var/lib/mysql/mysql.sock rw,
   /usr/share/mysql/charsets/ r,
   /usr/share/mysql/charsets/*.xml r,
   /usr/lib/mysql/plugin/handlersocket.so* mr,

You also need to reload apparmor profiles as the first google-link suggests:

sudo invoke-rc.d apparmor reload

And plugin works OK:

mysql> install plugin handlersocket soname 'handlersocket.so';

Query OK, 0 rows affected (0.01 sec)

mysql dev team against common sense

It looks like guys in Mysql dev team just have no real tasks.  Do you remember  that silly story "ENGINE=innodb" vs "TYPE=innodb"?

 Now the same story with replication settings. If you try adding master-host=<IP>  or any master-* into my.cnf in mysql 5.5 and above you'll get:
 [ERROR] /usr/sbin/mysqld: unknown variable 'master-host=my.masterhost.com'

WTF???

Because now you can only use CHANGE MASTER TO ...





  

Amazon AWS tricks as seen by noob

Every product has  a fine print today, just go to a local store and see. Amazon AWS is not an exception  and has some interesting, I would say, "features" that you should know about.

1) Each time you press Start or call start() from  API on instance you are IMMEDIATELY charged for the first hour. Hence if you make 10 star/stop during 20minutes you are charged for 10 hours.

Better example:

0:00 - start
0:10 - stop
0:20 - start
0:30 - stop
0:40 - start
0:50 - stop

Bill - 3  hours

2) You CAN run 64 bit image on Micro instance(t1.micro) but you CANNOT run it(64 bit image) on Small (m1.small) and Medium (c1.medium) instances. Do not expect  to upgrade 64bit Micro instance to small or medium, you can upgrade it only to Large and so on.



Well, these features are not game-changers but remember that it is better to keep instance running till the end of hour.

Bugs here

So again Network Manager got bugs, this time it is about high memory usage by nm-applet, obvious memory leak.
Again I wonder why NM is used in so many Linux distributions.  The only positive thing about NM is its integration with desktops but I would rather take working but bad styled software then highly integrated but very bad NM.   When my friends ask me "hey, what is nm-applet and why it takes 200Mb of RAM on my netbook?" I just tell them too install wicd instead of NM, also Ubuntu will still try to set NM back during upgrade to next version.

Amazon EC2 Endpoint URLs for HybridFox

hybridfox seems to have active development, at least you don't have to edit supported firefox version in install.rdf to get  it on Firefox 6+. Also people seem to use it more for Eucalyptus private cloud it can serve as 100% replacement for ElastiFox. With hybridfox everything looks the same but here is one big difference, it expects you to input  "Endpoint URL" (that is URL where Cloud API reside, it can be EC2_URL of Eucalyptus, for example).  I spent some time in search of these URLs for actual Amazon regions but found nothing better then just to steal the list from old ElastiFox


us-east-1  - https://us-east-1.ec2.amazonaws.com,
eu-west-1 - https://eu-west-1.ec2.amazonaws.com,
us-west-1 - https://ec2.us-west-1.amazonaws.com,
ap-southeast-1 - https://ec2.ap-southeast-1.amazonaws.com,
ap-northeast-1 - https://ec2.ap-northeast-1.amazonaws.com



PS. If you already had Elastifox installed, you can try Hybridfox, It will take all your preferences and settings.