The Open X adserver was the worlds leading open source ad server, boasting more than 150,000 installations. At present it is available as a hosted solution through OpenX, but the free downloadable script was sold out to Revive several years ago. Based originally on phpadsnew, this adserver has come a long way, and as revive now supports mobile and video ads.
One feature sorely lacking from the script is the ability to create separate advertiser and publisher accounts, but there is a commercial plug in for this available on the Revive adserver website: https://www.revive-adserver.com/
If you're happy with a hosted solution, then check out the OpenX website: http://openx.com/
Otherwise you can test out the downloads section where you will find not only Revive and OpenX but their predecessors, such as Max Adserver and Phpadsnew.
Another feature missing is a payment processor for publishers to be paid and advertisers to make payments, but the script is an excellent back end, and if you're handy with php you could create a front end yourself.
The main features are: standard banner rotation, click tracking, zone-based ad selection, zone-based campaign targeting, direct ad selection, ad targeting (per browser, domain, language, etc.), ad capping and support for Adobe Flash banners.
Open X started life as phpads, which later evolved to phpadsnew, then to OpenX and now to Revive, so it's had many years of dedciated development and is still going strong. For the Nostalgic few out there, it's worth looking at the original scripts, all of which are available for download here, but for those wanting user based support, then you're best off going with the latest release of Revive.