bpurcell.org - Blog
  1 2 3 4 5 6
7 8 9 10 11 12 13
14 15 16 17 18 19 20
21 22 23 24 25 26 27
28 29 30 31      

Subject Archives
Amazon EC2 (15)
ColdFusionMX (155)
Corvette (3)
Flash Lite (4)
Flash Media Server (5)
Flash Player (3)
Flex (39)
General Web Dev (14)
HDTV (3)
Jboss (1)
Jquery (2)
JRun (59)
Max 2003 (3)
Other (33)
PC Hardware (16)
Software (17)
SpatialKey (7)
Wireless (8)
Working Out (1)

RSS Feed
Feed Listing

Site Contents
My Blog
Wireless & Networking
Hardware & Gadgets
Software Picks
Contact Me

My Articles & Papers
Flex Performance
mm.com Under the Hood
Multiple Instances of CFMX
Multiple Instance Config
NLB with ColdFusion
Clustering CFMX for J2EE
Multi-Tier Hardware LB w CFMX
Cisco CSS & Coldfusion MX
JRun 4 Jini based Clustering
WiFi Growth

2ID Tacweb

Other Hobbys
Body-For-Life Challenge

Personal Projects
Family Pool
Deck Gate


Viewing by Category: Flash Media Server / Main
September 1, 2008
In recent years Flash video has taken over the web and the latest beta release of the player opens up a whole new set of features. Tinic Uro from the Flash Player engineering team has a must read post for anyone deploying video on the web. The latest Flash Player allows you to load and play .mp4,.m4v,.m4a,.mov and .3gp files using the same NetStream API that only played a much smaller subset of files before.

October 28, 2006

Since the debut of Flash 6 video has been available in the Flash Player. It has taken quite a while to catch on but if you are browsing around these days it is impossible not to stumble across a site that uses Flash Video. Sites like YouTube, Google Video, and MySpace have embraced flash video as their standard for delivering video to millions of users per day.

I am a common visitor to digg.com and I usually find most of the comments bashing Flash. Advertisers and ugly "skip intro" home pages have been the reason behind most of the bad blood. For video though Flash is hands down the best option. Flash is ubiquitous, universal and works across every common operating systems. It also is integrated with most of the gaming consoles that enable browsing.  I can't say that I always agree with John C. Dvorak but I definetily agree with a recent quote in PC Magazine. Unless you're streaming hi-def content to the TV over the network so that  you can watch IPTV shows, anything you plan on your computer should be Flash and only Flash. One of my favorite websites chicagobears.com has driven this point home for me. They have interviews and audio available on the site, if you attempt to click on the player UI you just get a broken image with no explanation. Most users wouldn't have a clue why it is not working. The video requires real player but without it you don't have a clue what is wrong. With Flash Player there are well-documented detection techniques that can provide alternate content and messaging to users that do not have the correct player installed. You just can't beat the experience you can provide as a developer. There are other options  (Windows Media Player, Quicktime, and Realplayer) each of them have the positive and negative points but if you compare all of the options across all OS's and browsers Flash definitely comes out the winner.

One of the common complaints with Flash video is full screen mode. With WMV content you can double click on the content and push it out to full screen. With a recent beta release of Flash 9 on labs.adobe.com 9,0,18,60 introduces full screen mode. This allows sites to deliver on the full immersive experience many people posting video to the web would like to provide for their viewers.

There is a lot of great content over at the Flash Video Devcenter  that can help you get started with Flash Video.


April 28, 2006

The following suggestions provide a good start to Tuning Flash Media Server 2.0.

Flash Media Server 2.0 Tuning

  1. Media caching control. Server.xml -> ResourceLimit ->FLVCache setting.

    FLVCahce -> MaxFiles = 1000
    Server is trying to keep any FLV or MP3 files up to this value in the caching memory. It will reduce I/O operation and get the content from the cache memory whenever it is possible.

    FLVCache -> MaxSegments = 500
    Each file in above setting will take this amount of segments to be kept in the cache. Each segment represents 128kb min and 256kb max, default size is 128kb.

    For prerecorded streams, there will be tons of different files in most of cases. Server will keep part of them in the cache when they are played first time. Expected total from base setting above will be 1000 x 500 x 128kb = apprx 64GB memory. That's why this default is not realistic honestly.

    For the optimal value, you need to check a couple of things. Here is an example. Let say you have 4GB memory in the box and allocate up to 2GB for FMS server usage. Anyway, Windows OS only allows up to 2GB per process, and if it goes above, OS will kill it and you will get fmscrash event. So it's better to control process doesn't use more than 80% of 2GB. Go back to sample, FMS server will use a certain amount for itself. Except that, I would say 1.2G for caching for example. And also I expect each play uses different file and 500 concurrent users expected. Then, I would set MaxFiles as 500. Then MaxSegments will be (1.2GB / 500 users) / 128kb = apprx 18 segments. In this given situation, caching will be best fit with following.
    FLVCache -> MaxFiles = 500
    FLVCache -> MaxSegments = 18

    As a second example, if site expects 500 simultaneous users, but they are probably watching same contents more often and it's about 200 different contents total. Then I could set it with MaxFiles=200 MaxSegments=46.
    As a third sample, if I want to cache a whole file content instead of part of it and each flu is avgas lamb long, MaxFiles=200 and MaxSegments=46 apprx.

    So you can expect server wouldn't exceed caching memory above 1.2GB range.

    Base line is you need to observe and define what is user behavior on the playback and set the optimal config for caching memory. If every playback ends up with new file, then increase file size for caching. If same contents are viewed more over and over again, increase segmentation to cache entire content.

  2. Script engine memory config : Application.xml -> JSEngine -> RuntimeSize
    RuntimeSize : 1024 ( 1 mb) default.
    If you use a lot of script objects in your FMS application, it could max out JSEngine memory. Actually, JSEngine memory is soft max, and it can go above that. When there are lots of load to try to grap more memory concurrently, then it could throw 'JSEngine out of memory error'. In general, this error can be resolved by increasing this RuntimeSize. If you use lots of FlashRemoting calls, XML, or XMLSockets etc, please consider to increase this value. Start it from 30mb (30720) and increase it with your load test result.
  3. Separate FMSCore process for each application or instance : Application.xml -> Process -> Scope
    Scope : (blank) - vhost scope.
    This scope default is vhost level. However, you can separate process for application scope or even instance scope. Instance scope could create so many different process files. Unless you need a specific need for it, I would suggest application scope at most. Since it is defined in Application.xml, it has a lot of flexibility. If it is set in global Application.xml, it will affect entire applications for example, <Scope>app</Scope>. After you restart the server, each application will have its own core process. Good thing of this is one application failure won't cause any other service interruption of other applications. Many cases don't need to separate them, but if you want to make an app isolated, then you could use this option. For example, one application has most of loads, but the others are not so busy. Then you can copy Application.xml and put into application folder which is same level of main.asc, and modify scope for app for the new Application.xml. After server is restarted, that app will be in new FMSCore process and all other apps are in another FMSCore process.
  4. Rollover FMSCore process for maximum stability. : Application.xml -> Process -> LifeTime
    LifeTime -> RollOver : value in seconds
    LifeTime -> MaxCores : value in numbers

    For example, FMSCore is still reaching about 2GB memory range in windows somehow, and it is killed. Or for some reason, there is any unknown interruption in FMSCore. If this happens every other day for example, then you can rollover fmscore daily basis with this config. Let say RollOver : 1200 and MaxCores : 3 for example. After about 20 min run, FMS will spawn new FMSCore process. New FMSCore will take over all new connections after it is created. Old FMSCore is still processing for existing clients with no problem. When all existing clients are detached from old core process, it will be collected eventually. MaxCores controlls there are not more than 3 cores to be created for same scope. For example, for an hour run, it could have more than 3. If there are more than 3, server will collect the oldest one. This rollover will help many scenarios without interrupting exiting services.

  5. File caching in Edge (Proxy) server. : Vhost.xml -> Proxy -> CacheDir
    enabled : false
    Please set this is true for file caching. FMS Edge server(Proxy server I mean here, please don't be confused with FMSEdge process) uses memory cache by default. For better performance it's better to turn on file cache together. If it on, FMS will save contents to the file system as well as in memory. When server need to read file, it will read from local file instead of asking from remote Origin server. Downside, it will fill up the local file system. Need to monitor the files and clear them out if it becomes too big.
  6. Multiple FMSEdge processes : Adaptor.xml -> HostPortList -> HostPort
    <HostPort name="edge1" ctl_channel=":19350">${ADAPTOR.HOSTPORT}</HostPort>
    When server plans to service tunneling traffic together, usually, you will add additional port like this. Fms.ini -> ADAPTOR.HOSTPORT = :1935,80,443. This should work fine with no problem. However, sometimes, tunneling traffic could be a factor for bad tunneling call like regular http request. If this causes any trouble, this would affect entire fms server because all ports are tied to one single FMSEdge process. If it is planned to serve different ports, it's recommended to separate them in different FMSEdge process. So it makes server more reliable from errors.
    <HostPort name="edge1" ctl_channel=":19350">:1935</HostPort>
    <HostPort name="edge2" ctl_channel=":19380">:80</HostPort>
    <HostPort name="edge3" ctl_channel=":19340">:443</HostPort
    This config will create three different FMSEdge processes. Note that <HostPort> tag 'name' and 'ctl_channel' should be unique in the server. Ctl_channel port is used to communicate between processes, fmsedge, fmscore, and fmsadmin.

Windows OS Tuning for FMS

  1. OS limit on sockets :
    Theoretical limit on number of sockets is 65536, however you cannot have this many on windows. On win 2000 you can get close to 25000, on win2003 this can go up to 30,000.

    Make sure the registry is configured properly :

    System Key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
    Value Name: MaxUserPort

    Data Type = REG_DWORD
    Data Range = 5000 - 65534 Port-Number
    Data Default = 5000

    Set this to 65534

    System Key: [HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters]
    Value Name: TcpTimedWaitDelay
    Data Type: REG_DWORD (DWORD Value)
    Value Data: 30-300 seconds (decimal)

    This helps OS collects TimeWait sockets faster and reuse them.

  2. Bandwidth : Nic cards can be 100MB/sec or 1GB/sec. This is their theoretical limit. In production it is recommended that you don't go over 50% capacity. This will take care of situations when there is a spike

  3. Memory : On windows a process can have 2GB of memory per process. Need to make sure that total memory used is below 2GB.
  4. CPU : A dual processor system is good for FMS. Windows networking suffers once CPU is constantly pegged over 50%. In production they should plan to have no more than 50 and have the headroom so that any spike can be handled well.

Thanks got to Woojin Choi for providing the details.

April 27, 2006

There is a new feature in Flash Player 8 that allows auto discovery of an Edge server on your local network without actually having knowledge of the server. When the connection is created a broadcast is sent on port 67 (DHCP port) and the Edge server answers the request and the Flash Player reconnects through the edge server. The following explanation was provided by Woojin Choi.

The feature in Flash Player 8 is called FPAD if you do not want to use it you need to set NetConnection.fpadZone = -1, to avoid any side effect. 

 Player 8 introduced UDP broadcasting capability with FMS2 release. This UDP broadcasting happens when NetConnection is trying RTMP connection.  Player sends UDP packet to the network which is called FPAD (named to differentiate from WPAD). If there is FMS on the network listening, it will respond to the request with proxy information. Player will receive the response, open it, reconstruct rtmp url with proxy information and try to connect fms through this proxy server(FMS Edge server). This feature is supposed to work with FMS AutoDiscovery function in Server.xml. This AutoDiscovery allows server to listen on FPAD UDP packets in the network and respond to it.

[FPAD config in client side]
 There are several configurations. They can be configured through ActionScript NetConnection object. Here is a code snippet with default value.
    var nc:NetConnection = new NetConnection();
    nc.fpadZone = 0; // default is 0.
    nc.fpadWait = 250; // in milliseconds
    nc.fpadOnly = false; // boolean. true/false
    nc.fpadPort = 67; // Target port to broadcast
    nc.fpadInfo.ip = edgeip; // only available if proxy is detected
    nc.fpadInfo.port = edgeport; // only available if proxy is detected.

Default behavior is that player broadcasts FPAD to port 67 in the network with fpadZone=0 value. These can be modified and should be modified before NetConnection.connect() is called.

  fpadZone = -1 disables this broadcasting mechanism. So if this isn't required, it must be set to '-1' before call connection.
  fpadOnly=true will only detect proxy in the network and won't connect to the server.
  fpadInfo object is available only when player detect the proxy in the network. You can trace it out in onStatus if proxy is available. Player reconstruct url based on this value fpadInfo.ip and fpadInfo.port.  If there is a misconfigured FMS for example, this information can possibly tell you why connection is not working.
  fpadPort can be changed to some other port if target FMS proxies are listening on different port other than 67.
  fpadWait is timeout for detection. If player doesn't get response back within this timeout from any FMS in the network, it will connect to FMS directly in nc.connect() information. For example, even if there is a FMS to listen on and it is slow to respond, player will still have a chance to connect directly FMS Origin. 

[FPAD in server side]
  FMS need to be configured to work with FPAD packet. It is called AutoDiscovery function. This only makes sense for FMS Edge server because FMS Edge can redirect the call, but Origin won't redirect it. If this is turned on in Origin server, client won't be able to establish connection.

  Server.xml -> AutoDiscovery -> Enable : true // enable autodiscovery
  Server.xml -> AutoDiscovery -> ProxyInfo : edgeserverip:port // ip:port for player will use to connect through. It is ip of Edge server and port edge server is listening for RTMP connection, which is 1935 as usual.
  Vhost.xml -> Proxy -> Mode : remote // enable server as Edge server

  Server.xml -> AutoDiscovery -> AllowZone : 0 // default allowZone is 0
  Server.xml -> AutoDiscovery -> MyZone : 0 // default MyZone is 0
  Server.xml -> AutoDiscovery -> BindInfo : :67 // 67 is default port FMS is listening, which is a well known port for DHCP

  This Zone information is very important for Edge server clustering. For now, it can be tweaked for client UDP broadcasting. It can be changed to any number for MyZone. For example, if MyZone is 8 and AllowZone is 8, it would allow only flash clients to broadcast to fpadZone=8. Default flash clients(fpadZone=0) won't be able to go through this FMS Edge server.

  FMS Edge server config has many benefits. One is to get better performance by distributing load from origin to edge servers. Second is to save bw for internet from intranet by caching data in Edge server. Third can help an organization to get around web proxy config when they can deploy FMS Edge in the network.  FPAD is useful when FMS Edge is launched in the network and clients can work seamlessly without knowing it.

Thanks Woojin for this posting.

October 31, 2005

If you haven't taken a look at Flash Media Server 2.0 (FMS 2.0) here is your chance for a quick overview. I presented two sessions at MAX 2005 on FMS 2.0, you can find the slides below both in adownloadable PDF. FMS 2.0 retains the same functionality available in Flash Communcation Server 1.5 but add many new exciting features.

FMS 2.0 introduces new scalablity concepts through the use of Origin and Edge servers. The new architecture provides the optimal deployment for large scale media applications, simplifying load balancing, failover and clustering. Scale with multiple Edge servers to manage bandwidth, traffic and processing while giving the Origin server a single view to reports, logs, media assets and application logic. FMS 2.0 improves reliability by running the server in multiple processes. In FCS 1.5 the server contained only the admin and server processes with only two processes if an application causes stability problems all applications would fail. FMS 2.0 introduces multiple processes allowing administrators to segment VHosts, Applications or individual application instances in their own processes.

There are several video improvements with FMS 2.0 as well.

  • Iimproved video codec using On2 VP6 with Flash Player 8
  • AutoStreamTypeSwitching - allows developers to stream either VP6 or Sorenson encoded video depending on the version of flash player

As a support engineer for FCS I felt a lot of pain from the logging system since it logged data to FLV files. I can remember running the log reader on 300MB log files. FMS 2.0 introduces text based logging based on W3C format so you can use third party log readers to parse through the logs and build reports.

New developer features

  • File class enables writing to the servers filesystem
  • XML Object
  • Web services support
  • Integrated SSL
  • Access DLL
  • Java SDK

Flash Media Server 2.0 also introduces a new administrative console that is worth taking a look at. Download the PDF of my presentation for more information. You can also join the public beta.

Page Render Time:250