#rabbitmq IRC Log


IRC Log for 2011-07-16

Timestamps are in GMT/BST.

[2:30] <agronholm> can someone enlighten me about the purpose of channels?
[2:30] <agronholm> rabbimq's docs don't really say what they're for
[2:33] <antares_> agronholm: AMQP connections are multiplexed with channels
[2:33] <agronholm> I get that much
[2:34] <antares_> agronholm: basically, apps do not need to open N connections, instead, they typically use 1 connection and as many channels as they need
[2:34] <antares_> agronholm: the rule of thumb is "1 channel per thread" or "1 channel per component". Many apps can get by with just one channel.
[2:34] <antares_> agronholm: this makes AMQP more "firewall friendly"
[2:34] <agronholm> mmkay
[2:34] <agronholm> thx
[2:35] <agronholm> I suspected something like that
[2:35] <antares_> agronholm: also, certain protocol exceptions are per-channel
[2:35] <agronholm> but then pika's docs say connections shouldn't be shared between threads which does not really fit into this channel-per-thread model
[2:35] <antares_> agronholm: it depends on the client, in the Java world "one channel per thread" is very common
[2:36] <antares_> agronholm: when in doubt, try using just 1 channel first
[2:36] <agronholm> yup, rabbitmq's official java client is probably the one I'l end up using
[10:26] <siovene> Hello. I'm working on a website that uses django + celery + rabbitmq to perform some pretty CPU intensive tasks. I have realized that running this tasks on the same server that serves the website is suicidal.
[10:27] <siovene> Can anyone point me to instructions that explain how to setup celery and rabbitmq on a multi-server (ideally EC2) architecture?
[10:27] <siovene> Specifically, I need to know how to perform the data exchange (i.e. do I need to write webservices myself or would the various RabbitMQ instances abstract the communication for me?)
[13:37] <PiotrSikora> siovene: RabbitMQ is the messaging layer, you don't need to do *anything* for your setup to work on multiple machines
[18:49] <joevandyk> I installed rabbitmq on osx with brew. I'm getting this error: https://raw.github.com/gist/1086587/0d93e46448b312db39d47a24d4b28765212df4de/gistfile1.txt
[18:49] <joevandyk> I'm using cloudist in ruby to submit jobs to rabbitmq
[18:49] <joevandyk> I'm using rabbitmq 2.5.1
[18:50] <joevandyk> sustrik: have you looked at cloudist?
[18:50] <antares_> joevandyk: are you upgrading or is it a fresh installation?
[18:50] <joevandyk> antares_: fresh installation
[18:52] <antares_> joevandyk: it looks like one of the modules, the tracer, new in 2.5.0, cannot load
[18:52] <joevandyk> how did you figure that out?
[18:52] <antares_> joevandyk: I have 2.5.1 on OS X from homebrew, too
[18:52] <antares_> by reading the crash report?
[18:52] <joevandyk> lol
[18:52] <joevandyk> i did too
[18:53] <joevandyk> i did not get that answer :)
[18:53] <antares_> {rabbit_trace,init,1}
[18:53] <joevandyk> antares_: thanks -- i turned off tracing
[18:54] <joevandyk> i thought a guest user was supposed to be automatically installed?
[18:54] <joevandyk> with full permissions to '/'
[18:54] <joevandyk> on my system, there were no users
[18:54] <antares_> joevandyk: it is pre-seeded, yes
[18:55] <antares_> joevandyk: that means initial DB seed did not happen for some reason
[18:55] <joevandyk> antares_: hm
[18:55] <joevandyk> antares_: how can i get that to happen?
[18:55] <antares_> and that's also why tracer did not locate pre-declared exchanges
[18:55] <antares_> joevandyk: try this
[18:55] <antares_> joevandyk: kill rabbitmq if it is running
[18:55] <antares_> open a new shell tab and run rabbitmq-server
[18:55] <antares_> it will print all boot steps
[18:56] <antares_> this is how I always run it (mostly to test amqp gem & apps w.r.t. broker crashing, it is very easy to imitate by killing rabbitmq running in the foreground)
[18:57] <joevandyk> antares_: should i delete the database dir?
[18:57] <antares_> joevandyk: not sure, check out the log. May be a file directory permission issue, for example. I keep Homebrew under /usr/local that is owned by my effective user.
[18:58] <antares_> joevandyk: yes, please do
[18:58] <joevandyk> hm, it works now. thanks!
[18:58] <antares_> joevandyk: it should be under $HOMEBREW_ROOT/var/lib/rabbitmq/mnesia/
[18:59] <antares_> joevandyk: so maybe homebrew formula is broken and sets up permissions incorrectly
[19:07] <joevandyk> antares_: do you have any recommendations for a simple async/rpc library for rabbitmq with ruby?
[19:07] <joevandyk> btw, thanks again for your help -- i would have gotten stuck on the trace problem
[19:08] <antares_> joevandyk: I only use amqp gem
[19:10] <antares_> joevandyk: http://rubydoc.info/github/ruby-amqp/amqp/master/file/docs/Queues.textile#Objects_as_message_consumers_and_unit_testing_consumers_in_isolation demonstrates how you can integrate amqp entities into OO Ruby code. That's all I use and we are adding some higher-level abstractions to the amqp gem itself, although right now it is not really a high priority.
[19:14] <joevandyk> antares_: i would love to see some higher level things in amqp
[19:19] <antares_`> joevandyk: me too. But after 0.8.0 release. We have a bunch of things already designed, for example, AMQP::Pipe
[19:20] <antares_`> joevandyk: http://rubydoc.info/github/ruby-amqp/amqp/master/file/docs/PatternsAndUseCases.textile is what I am going to start with (docs + examples), some patterns will certainly benefit from additional primitives in the library
[20:12] <joevandyk> antares_: i started roger_rabbit https://github.com/joevandyk/roger_rabbit a while back
[20:12] <joevandyk> it would be nice if the amqp provided an api that simple
[20:15] <antares_> joevandyk: it is difficult to both provide access to all the AMQP features & rabbitmq extensions and be the simplest messaging solution available
[20:15] <joevandyk> :) i figured, but it would be nice to have a few methods that provided sensible defaults for the simple case.
[20:16] <antares_> joevandyk: I am not ruling out a very minimalistic API like this being part of amqp at some point, but we have been there, done that with amqp 0.6.x releases that used implicit channels
[20:16] <antares_> joevandyk: one very difficult problem APIs like roger_rabbit completely ignore is application recovery
[20:17] <joevandyk> antares_: yes, it does nothing yet. i still haven't understood everything about rabbitmq.
[20:18] <antares_> joevandyk: amqp 0.8.0.RC14 will have a very simple solution for automatic recovery (you just pass one more option to Channel constructor). But it was only possible because we did not try to hide AMQP entities.
[20:19] <antares_> joevandyk: so we will see. I already pulled a few good ideas from the qusion gem into amqp. I just try to only add solutions to problems I see over and over again.
[20:20] <antares_> joevandyk: this is more or less how amqp gem is developed (although I am not nearly as smart as Rich Hickey): bit.ly/hammock-driven-development
[20:21] <joevandyk> antares_: for example: the request/reply pattern. https://img.skitch.com/20110716-1rdr1rwkaf2qi6u5cceuudewkr.png If that's a common pattern, it would be nice to have that code inside a method.
[20:22] <joevandyk> Then you could do something like AMQP.rpc('routing-key', 'message-json') { success { |result| # something } }
[20:23] <antares_> joevandyk: there was AMQP::RPC in 0.6.x days. In fact, it is still there. It is super-magical and even redefined BlankSlate to be "simple". The end result? It broke builder 2.1.2 and some Rails apps badly
[20:24] <antares_> joevandyk: also, because amqp gem 0.6.x invented its own terminology (MQ instead of Channel is one example), amqp gem users spoke their own language on the rabbitmq mailing list. No one was able to understand what they are talking about.
[20:24] <joevandyk> https://gist.github.com/1086659 -- did it work something like that?
[20:25] <antares_> joevandyk: and because everything was implicit and hidden by "simplicity layer", users had no idea what other rabbitmq users suggested to them :(
[20:25] <antares_> joevandyk: let me put together an example
[20:27] <joevandyk> antares_: updated https://gist.github.com/1086659 There are a bunch of libraries that seem to provide a simpler wrapper around amqp (cloudist, minion, sweat_shop, beetle, etc) -- having something standard would be great.
[20:27] <antares_> joevandyk: https://gist.github.com/1086664
[20:27] <antares_> joevandyk: the same object was used for requests & replies
[20:29] <joevandyk> antares_: hm, that doesn't make any sense to me.
[20:30] <antares_> joevandyk: I agree that our of 4-5 "syntactic sugar libraries" on top of amqp we better recommend one, but I am not sold on the idea of shipping it as part of amqp itself. That said, I am also not against it.
[20:31] <antares_> joevandyk: wel, that's the point. Something that looks simple to one person doesn't make sense to some other people. And if it hides all the AMQP entities, it becomes very difficult to understand even to people who have been using AMQP for a couple of years.
[20:32] <antares_> joevandyk: my recommendation is this: build something you would like to use, if many people like it, I will add references to it to the amqp gem docs and we can move it under github.com/ruby-amqp
[20:33] <sid_> Could someone direct me to example on how the AMQP gem is used with Rails. I am trying to understand how to dynamically generate queues and subscriptions
[20:33] <antares_> sid_: you typically do not need to generate queues dynamically
[20:34] <antares_> sid_: you declare them once on boot, then use them. If you need to route messages differently, often routing key is what needs to change.
[20:36] <joevandyk> antares_: what if the queue goes away? (i.e. a crash)
[20:36] <antares_> sid_: http://rubydoc.info/github/ruby-amqp/amqp/master/file/docs/Queues.textile#Objects_as_message_consumers_and_unit_testing_consumers_in_isolation and http://rubydoc.info/github/ruby-amqp/amqp/master/file/docs/ConnectingToTheBroker.textile
[20:36] <antares_> joevandyk: if broker crashes, the queue doesn't necessary go away
[20:37] <antares_> joevandyk: durable queues will be there after reboot. But if broker crashes your app needs to do more than generate a queue name.
[20:38] <antares_> joevandyk: this is documented (not in a lot of detail, but hopefully reasonably well) at http://rubydoc.info/github/ruby-amqp/amqp/master/file/docs/ErrorHandling.textile. amqp 0.8.0 will ship with so-called "automatic recovery mode" that reopens AMQP connection & channels and then redeclares entities & consumers for you. For "the 80%" it works surprisingly well.
[20:38] <sid_> antares_: thanks. But does that mean you would run the AMQP code as a separate application and not with Rails
[20:39] <antares_> joevandyk: and it is just 1 argument you pass to AMQP::Channel#initialize. Not very difficult to use ;)
[20:39] <antares_> sid_: AMQP is a messaging protocol. Its whole purpose is to let multiple applications work together.
[20:39] <antares_> sid_: for in-process communication AMQP probably doesn't make much sense. What are you trying to do?
[20:40] <sid_> I was trying to build a chat system something I would normally do with XMPP and ORbited/websockets
[20:43] <antares_> sid_: I see. You certainly can use AMQP to publish messages received from Web clients
[20:44] <antares_> but with Rails specifically, notifications arrive asynchronously and to push them back to Web browsers you need to use Goliath or something similar
[20:44] <antares_> sid_: have you seen travis-ci.org?
[20:44] <sid_> antares_: No I will check it out now
[20:44] <antares_> sid_: it distributes build requests to workers one way and progress reporting to Web browsers using Pusher.app
[20:45] <antares_> sid_: and though travis doesn't use amqp today, we are working on that. In your case the harder part is distributing messages back to Web browsers.
[20:46] <antares_> that has to be done with something like WebSockets but for everything else, you can use AMQP
[20:46] <sid_> antares_: I dont think i would have trouble pushing messages to the client, I have that part figured out but I would like to know how to manage users to subscribe to queue. I understand how its done in a simple Ruby app
[20:47] <antares_> sid_: also, there is https://github.com/ruby-amqp/amqp_stomp_interoperability_example_with_ruby
[20:47] <antares_> sid_: I have an example chat app that uses AMQP and WebSockets, actually
[20:47] <sid_> antares_: But when working with a Rails app where users are dynamically created I would like to have them subscribe to a feed and receive messages on creation
[20:47] <antares_> sid_: it was even stress-tested by Reddit
[20:47] <joevandyk> for what it's worth, I think I like the Beetle API the best.
[20:48] <antares_> sid_: https://github.com/videlalvaro/rabbitmq-chat
[20:48] <sid_> antares_: Could I take a look at how it works if the code is available publicly
[20:48] <sid_> antares_: Awesome thank you
[20:48] <antares_> sid_: it is not in Ruby but sounds exactly what you want
[20:49] <antares_> sid_: WebSockets + RabbitMQ + a RabbitMQ extension to keep recent messages around for newly connected clients
[20:49] <antares_> sid_: http://videlalvaro.github.com/2011/05/rabbitmq-chat-post-mortem.html
[20:52] <sid_> antares_: Ah looks like erlang. I did look at the second link recently.
[20:56] <antares_> sid_: it is Erlang but as far as AMQP usage goes, you can reuse many pieces of the architecture
[20:57] <antares_> sid_: at least this is something that is very similar to your case & open source. And also handled 5K delivered messages per second.
[20:57] <antares_> with < 90 megs of RAM
[21:01] <antares_> sid_: if you will decide to not use AMQP, check out zeromq.org. It is different but very good, too.
[21:03] <sid_> antares_: I did look at ZeroMQ. What would you say is the ideal use case for ZeroMQ?
[21:07] <antares_> sid_: apps where there is no "broker" or special nodes, like Skype or other p2p systems.
[21:08] <antares_> sid_: also, in-process and inter-process communication on the same machine. there are actors frameworks built with zeromq
[21:09] <sid_> antares_:Ah, file sharing? Brilliant. Thank you very much. Its been awesome and extremely informative
[21:11] <antares_> sid_: :) cool. If you end up going with Ruby & AMQP, amqp gem docs are at http://bit.ly/amqp-gem-docs (they describe pre-release version but it is more solid than the released ones)
[21:13] <sid_> antares_: Thanks. Ill try amqp for a bit before I try ZeroMQ.

These logs were automatically created by MyxoRoboto on irc.freenode.net using the Java IRC LogBot.