This is a presentation by Jared Mauch of how he managed to dig and build his own fiber network in his town without help of big telcos.
He shares his lessons learned (if you feel inclined to build your own) and details his expenses (it's not cheap, but not impossible either). It's a very informative and fun talk!
So this is not a case of somebody who had more enthusiasm than experience attempting to do a rural broadband project. This sort of thing does crop up commonly in attempts at rural WISP projects, where somebody who's never had 'enable' on the routers of any ASN decides to build their own ISP.
In this case it's somebody with the great depth and breadth of knowledge to do it, dissatisfied with their local status quo for residential last mile service, deciding to implement it themselves.
How much of this presentation could be applicable outside of the US? As in: How much of this is general knowledge that you can apply if you're trying to achieve the same goal and happen to be situated outside of the US?
There are many parts that posed challenges. At the height of covid lockdowns - talking to contractor about if they will start at all was a big decision. Thankfully they were a family business and the work is outdoors in a rural setting so lower risk.
I asked this question in a thread on a different story, then when I saw this thread I immediately thought you would be the perfect person to ask this question:
Have you ever found a way to correlate or list rural areas in the USA that already have high-speed FTTH? A lot of people here would be interested in that information, since they plan to move out of big 1st, 2nd, or 3rd tier cities and will be working from home indefinitely.
Very few ISPs publish service area maps, never mind accurate coverage maps. The only public data about broadband availability is based on FCC 477 forms, which is notoriously inaccurate. It's so bad that the FCC recently had to walk back claims of broadband coverage for 63 million households. The FCC data is so terrible that you can only use it to rule out coverage areas.
To get any kind of reliable data you'd have to manually try to collect information scattered all over the web. Go over ISP websites, the list of muni broadband networks, carrier portals,...
Even so, ISPs can and will readily deny service, even if they at first say that there is coverage. The only way to make sure is to order service and see if they deliver.
If you have the budget for it, you could consider using commercial services like Fiberlocator, but they mainly cater for the commercial and wholesale markets. I don't know if they cover FTTH. They may, if the network in question also offers business service. They have an API, but that's probably not included in the yearly subscription fee of just under four grand.
our local area had to hire someone to drive all the roads and look for the telltale signs of the utilities to get an accurate map. This was taking a cross-section of a survey that went out to everyone and the FCC477 data that is filed. The accuracy in the past has been at the census tract level, which shows my area as served as someone in the census tract has service from Comcast.
Here in my 3rd world country, fiber is getting pretty common. Small ISPs just rent space on poles and lay the fiber to the extend that I can choose from 4 different fiber ISPs in my home.
It amazes me how difficult it seems to get the same in more developed places like US/Canada.
I once looked into running a fiber cable on poles for 2,000ft. It has to be done by certified professionals. I got quotes from $25,000 to $75,000. Just to hang a wire on what would probably be 20 wooden poles.
The pole owner controls what is attached.. There may be pole attachment agreements you can review, but depending on the distance and complexity of the route and how many poles may need upgrades the costs still sound a bit high, but maybe they're all much taller poles. A pole itself costs $800 to start for a 32' delivered, but you need someone with a truck crane, which also requires a crane license to set it. Figure a few people at $25-35/hr as well to prep and repair the ground and you can see why it's a big cost.
I'm a fan of one touch make ready rules, but when you get to rural areas many of the poles are old, and to get the 18' clearance on a road span requires moving everyone else up, which may mean a new pole.
that's what i did. there also are not poles the whole way, so with some drilling, it's easier to just drill the whole way. plus if someone digs it up, i can go after them for repair costs.. hard to go after a tree for a wind/ice storm.
I sometimes think the US functions as a giant experimental ground for the rest of the world to learn in which domains a combination of capitalism and devolved regulatory authority (which seems to drive rapid innovation) works and also DOESN'T work :-)
We've had fiber to home for years now in the East. Largely driven i think by governments seeing this was one area where it didn't work.
The politicians are paid off by big Telcos looking at you ComcastNbcUniversal. So much so cities and suburbs have laws on their books that they deal with only a single internet provider. Politicians and their empty talk of ‘competing with other countries in terms of technology infrastructure’.
Cronyism exists in all political systems. Doesn't matter whether it is a monarchy, dictatorship, socialist democracy, federal republic - it's always a concern.
But, cronyism is not capitalism. It's actually a subversion of capitalism.
I recall reading somewhere where Australia has a system where the government owns/runs all the common network infrastructure, even to the home, as a for-profit entity and then allows any entity to rent and provide services on the shared infrastructure. This seems brilliant to me. I think its called https://www.nbnco.com.au/
I mean the original idea was fantastic - the network infrastructure was going to be fibre to every household in Aus (with a few exceptions for ultra-remote locations) - 1 gigabit down/up.
Then the conservative gov opposition ran a campaign (bolstered by incumbent telcos and Rupert Murdoch of Fox fame) that fibre was unnecessary and it'd be much cheaper and quicker to provide a patchwork mix of different modalities, the idea being that fibre would never be necessary.
The reality is that the opposition just needed to run on a campaign that was "not what the Gov wants to do".
Long story short, opposition won government, and ended up buying back (at a premium) a whole bunch of old copper infrastructure from the major telco whom they'd sold the infra to decades earlier. The costs of mismanagement and using such a patchwork mix blew the whole project out and here we are 10 years later with the "quicker cheaper" STILL not finished rolling out.
Where it has been deployed, it's suffering under loads that it can't keep up with - I'm currently getting 1.5 Mbs down 0.72 up.
Plus we spent $31 Billion more than Abbott/Turnbull told the voters it would cost.
We would have saved $11 Billion if the original FttP rollout, which had just begun, was allowed to proceed.
The valuations of Murdoch's decrepit coax network and Telstra's ancient copper network have skyrocketed, now that they have been forced back in as the last mile delivery to the home.
I read recently that given better pricing of fibre rollouts nowadays, it would take $7 Billion to rectify this failure for most of Australia, by completing the FttP network that got cancelled.
Political lies and/or gross incompetence resulted in spending 20% more than fibre to the home, and ending up with unreliable much slower copper to the home, and harder to maintain mish mash of complexity via grab bag of mixed technology in use now.
The politics of the NBN were an absolute travesty. It made me realise that as much as we make fun of the US and their orange baffoon, our politicians are far worse because they are competent at their corruption.
I did get one tiny bit of schadenfreude from the whole thing: I had lengthy debates with a coworker about whether the NBN is worth it or not. He said it's a waste of money. My position was that even if it improved the economy just 1%, it would pay for itself in no time at all.
Now, in the middle of the pandemic where everyone is working from home, I have gigabit NBN fibre-to-the-premises. He's on 2 Mbps ADSL v1 with 10% packet loss.
While I'm a fan of the single open network with multiple access model, I really wouldn't use nbn as an example of a government provided network, because of the politicking that went on during it's inception, the corruption that went on during it's build, the changing of horses midstream from fibre everywhere to a mix of less-capable technologies, and the way it was effectively built to be sold to the incumbent.
If you want to see how to build a national broadband network, look across the ditch at what the UFB gets you. AUD$89 for 100mbps, versus NZD$85 for 1 gigabit.
(Orcon in NZ, and TPG used in AU for comparison purposes)
It is hilarious to me as an Australian that anyone would call the NBNCO out as a 'brilliant idea'.
In theory yes but the 'national broadband network' has been nothing but a giant political mess and used as a tool to misinform and manipulate voters. It has resulted in ridiculous overblown budgets and a subpar below global standard product. I live 7km from Sydney CBD and I still have ADSL2 with 9mb down and 200kb/s up which gets 75% packet loss anytime it rains. The NBN was established/announced in 2009, it’s now 2020 and I am still waiting..
Anyone who is under the impression that the Australian government plans to maintain ownership has their head in the sand. This is a paid for tax payer network infrastructure project which will be packaged and sold off to private business in 5 years time so that private business can reap the rewards for the long term. This is what happened to our previous common network infrastructure provider Telstra.
It’s this way because of how regulated everything is. You have to jump through hoops to get permission to do anything here in the U.S. in other countries you just do it.
It's fairly regulated here. The regulation says that the companies that own the poles are obligated to sell that space to any other telcos that want to rent it.
It would be interesting to see someone just do it in the U.S. as well. A couple of guys with some amber flashing lights on their trucks. One with a bucket truck, another with big spool of fiber; guys with hard hats and safety vests. Cops aren't going to stop and ask questions. Maybe a telco guy driving by might ask a supervisor if they even cared enough about it. Just have guys that can actually do it correctly so the finished work doesn't look suspicious.
Once it is up, the regulations will work for you. Some day, some other "legit" installer will come by and look at the cable and basically shrug their shoulders at it, and then do their work around it. It would take weeks if not longer to find out nobody knows who owns it. Nobody's going to take the time to trace it back. And I'm guessing nobody is going to just take it down for fear of taking the wrong thing down. Maybe they cut the line and wait around to see who comes to splice it?? It would make for an entertaining story.
A municipality local to me (New Lebanon, Ohio) "just did it" and hung 150 banners honoring local veterans from power poles. The local monopoly power company threatened the municipality with >$300 per-pole hanging fees and demanded they be removed. There was some back and forth but eventually the power company gave-in to public sentiment (honoring veterans and all). I would suspect "just doing it" would put you on the business-end of civil, if not criminal, litigation.
I have done this! It's harder/more stressful than you might think, but it's absolutely possible. We installed relay radios powered off the photocell plugs to bring low-speed internet to a family friend's farm, so the actual installation doesn't even involve wires between poles, just shims under the streetlamp photocells. Presumably they'll get knocked out when the streetlamps are upgraded... but it's been six years so far with no issues.
the issue with this approach is if you depend on it, it may just go away some day, either with the police called and if they track it back to you, you may be prosecuted under the laws that protect utilities. These are less forgiving than you might think.
It's a big deal to damage someone elses infrastructure, can result in bills and lawsuit to reclaim losses. If the pole owner notices, they may do something (or just ignore it). What you don't want is them tearing down your stuff without notice.
> Maybe they cut the line and wait around to see who comes to splice it?
I can't pull them up now, but I've seen a number of credible stories on the internet of Comcast and AT&T technicians going to homes to do service and then intentionally cutting lines of the other company that service the building or nearby houses. Obviously this isn't super common or it would be bigger news, but it definitely happens.
I feel that having fiber in big cities is a given nowadays. I don't think telcos have any plan on supporting those smaller remote towns with fiber. Maybe investment does not pay off in a reasonable amount of time.
Fiber may not be the only option though. Starlink should be becoming available soon in North America. Only time will tell if that will work out though. WiMax, for example, was a failed hard.
I'm was born in small-town in Russia with 5k people. There was an area with private houses and 2-3 floors houses with 18-24 apartments. We have linked attics of buildings with apartments with cat 6e cables and cheap D-Link switches. In 2002 our telephone hub was replaced by a digital telephone switch station where we were renting five 2 Mbits ADSL channels. So users were connected to switches in attics for a small price and anyone who wants access to the internet could configure VPN to our FreeBSD "router". I don't remember the speed limitations, but the cost of 1 Mb was about $0.04(2 Russian rubles). In 2004 ADSL was upgraded to ADSL2+ with 20 Mbits per port. In 2006 all copper equipment was replaced by fibre channel cables/routers. And some people from private houses connected to our network(they were using dial-up). In 2010 huge Russian ISP Rostelecom bought our small network. At the end of 2006, I moved to Moscow because I went to college. And of course, me and 3 of my new college friends has built local ISP in university's dormitory :) The new dormitory was opened after two months after the start of the school year, so there was no any network. Officially it was not possible to connect this building to the internet. We rented a Wi-Fi router in someone's apartment in the building next after our dormitory. I don't know what happened with network because I was expelled from university :)
It's definitely not a given even in big cities. In San Francisco the only places I've seen it offered are big apartment buildings with 50+ units. At least from what I've seen, even in pretty dense parts of the cities if you're in a small apartment building with like 10 units or you're in a duplex or attached SFH you can't get fiber service.
The upside is we can still get gigabit service over cable lines through comcast for a fairly reasonable price (although with terrible upload speeds, data caps on the plan, and the need to fight with support every year to prevent your price from going up, as with all comcast plans).
Way way off. North America is way behind in terms of residential internet speeds even in the most dense areas like NYC and San Francisco etc. Sonic just recently started ramping up around the Bay and if you're lucky enough to have service its great. Still, probably 90% of people in the Bay and 75% of the city are stuck with 30 Mb/s up max from Comcast.
Starlink won't be high a replacement for fiber for a decade if we're being optimistic. These projects have huge ramp ups, tons of expensive launches, especially if they're successful, tons of users means slower speeds for everyone. And that all assumes success.
It's actually cheaper to install fiber than POTS or Coax. Little remote mountain towns with < 100 homes have fiber when they barely had POTS (and no way was it going to support ADSL). These guys went from 28.8kbps to Gigabit in one jump. It just took a few decades longer at dial-up than the rest of the country
Go look at the isps doing last mile FTTH with $700 splicers and roof to roof fiber in central Rawalpindi, Pakistan for instance. The USA or Canadian construction methods are much more organized and engineered, but also costlier.
Being on the poles can be more expensive than underground. Trees, cars and even bullets can damage the wiring. Underground is more expensive to start with but cheaper long term.
Underground has a bunch of other issues as well, in the UK all records of the underground ducting have been lost in many areas in the 50-70 years since it was installed, which makes it a big headache.
Which makes fixing, replacing etc quite expensive and the quality can be suspect.
It only amazes if you think everything in first world has to better than third world. And IMO it is about as immature as first worlder who thinks that everything in third world has to be crap.
There are some Facebook groups around with folks doing this kind of thing. The one I follow is "WISPs turned FISPs" with lots of folks building FTTH networks on small scales. (not sure how to link directly to a FB group.)
Also shameless plug for my website startyourownisp.com which has a similar theme.
Where did you end up running the fiber to, and how did you find out about this nearest connection point? Thank you for sharing your experience, very inspiring.
I had to build it 2 miles north to a fiber rich intersection. Even with it there it doesn't mean anyone will sell to you. I kept getting results a mile east or west of where I was going. Things like fiber locator or infrarpedia can help you figure out who is there.
Cool sutff, I wish I had the capital to do something like this. Although given how hard the existing ISPs are fighting against municipal internet here I wonder how I'd fare.
IME it's rarely (if ever) pushback from incumbents that kills a small ISP. Generally it's all the same kinds of things that any other business struggles with - making your product something people actually want, letting people know it's available, finding the right employees, etc.
Local regulatory barriers exist and if you aren't persistent you can't overcome them. There were many times where I wondered if it was a bad idea or .. what am I doing here.
Being a telephone company made it easier to permit and working with the groups to permit it became possible.
Just watched most of the video, and holy crap, this is super cool. Jared got frustrated by the (very slow) internet options available to him, and started his own isp to solve the problem for him and his neighbors.
A few years back I knew a guy who built a network for a few neighbors South of Ann Arbor. I was on some discussion group and just coincidentally CmdrTaco was complaining about his poor internet access South of Ann Arbor. Chimed in to provide an introduction to my friend and together they expanded the network.
Wish East Lansing had all the independent efforts that Ann Arbor does. Been trying to get fiber here for ten years, none of the local providers will come out to the suburbs ;<(.
1) He showed that he can achieve 700ish Mbit/s at his place. Does it mean that the 20 subscribers of his service are sharing this? If he wants to increase the overall throughput does he have to place more fiber cables?
2) Did he use any sort of repeaters to deal with the attenuation of the signal?
can confirm the shape of this assertion. Having a 10G even if rate-limited to something just over 1G gives the burst. When my son got a new laptop pre-school starting we could see those updates and game downloads on the graph, the other homes were just noise.
I believe he said he's paying for 1.5 Gbps capacity over a 10 Gbps uplink (though it's currently only set at 1 Gbps and needs to be fixed). So the maximum shared bandwidth is 1.5 Gbps, but he has physical headroom if he needs more.
Great talk, enjoyed it immensely. Say you wanted to increase the bandwidth of the line, would this simply be a matter of running the another length of fiber through the existing conduit?
in the video Jared said he's 25 years in the industry. This being nlnog, I guess in networking, one layer above the physical? good mechanical engineering skills also helped with building an diy conduit blower for the fibre cables.
The thing I most wonder about if you start to be a service provider and need to provide ongoing connectivity - you need a backup plan for when you're out of town.
And he didn't actually reply to the question on how much of his own time he has spent doing all of this.
It's impressive, but, at the same time, I'd imagine it's way too stressful to run by yourself in a small town like that. Without really that much payout, money-wise.
Are you saying you're going to recoup the value of your time in under 4 years, or would you just recoup the initial non-time investment in under 4 years?
It may sound like you might be spending something around 20 hours per week on this, if not more. If you bill your own time to your new ISP at market rate, when will you recoup the investment? Is that still 4 years?
To put some math into perspective, (20 hours/week * 4 weeks/month * 100 USD/hour) / 50 USD/month/customer = 160 customers -- that's how many customers you need to cover just your time alone, without considering any other expenses. The math should definitely work out given an adequate intake, but I think it's also easy to get lost in the community and tech part of it, forgetting the business case. Especially if you live in the middle of nowhere, without that many neighbours to sign up.
Even if Jared is spending 20 hours per week on this project now, that will not be the case after construction is finished.
As such your calculation does not hold and 160 customers will not be needed to recoup costs, cash or non-cash. There need only be enough customers to cover running expenses, administration costs and once off investment costs over time.
This is inspiring. Its amazing that he was able to bust this out for his community. Its mind blowing the amount of perserverance.
I really want to break free from my job and build something of my own. Like this. Really inspiring to see someone that had a metaphorical mountain between them and a goal, push through. I cant imagine anything I will be doing, will be nearly as problem ridden as this project.
Shows what confronting one problem at a time, and not giving up, can result in.
What happens when Jared sells his home or has to move? Everyone has fiber wired into his basement for connectivity. Would the new homeowner be obligated to continue running this?
Personally if I were a neighbor I'd still sign up with this risk given the ISP options available, but would still be a concern. Your Internet service and home value depend on this guy keeping things running "forever".
Most likely, the equipment would move to a community owned and maintained equipment enclosure/shed. This is a worn path for coops and other endeavors like this where residents go in to buy a DSL DSLAM and pull fiber to it, with the "last mile" being copper runs.
It's pretty easy, I just sell myself an easement on the property for $1 before I sell to someone else. I can also just move the electronics somewhere else. The nice thing about fiber is the range, I can be within 25dB of here and it will still work. With average loss of 0.5dB/km that gives you a decent range
@jaredmauch Would you still have considered doing this today with Starlink now happening?
I stopped the video once the interviewer portion started but how does someone justify $125-150k investment to become an ISP? I'd maybe consider this if I owned a large chunk of land out there and planned to sell it and then sell the internet subscriptions to individuals who build there.
I basically don't know much about the nitty gritty of network engineering but this was very satisfying to watch. Especially the anecdote he shared of a customer going "holly, molly" after he started using this service. Very nice.
On youtube, you can click on the ... and click the "open transcript" link to access an auto generated transcript. Haven't checked the transcript of this video, but in general they've become quite good over the years.
I can't imagine taking all of that on if someone else was paying for it, let alone financing so much of it yourself -- not to mention all the sweat equity you obviously put into it.
So many people in the States have little to no choices on ISPs. Your story is very inspring and encouraging.
I hope you continue to reap the rewards of your hard work for many years to come.
I ran short distances (like my private road, or driveway crossings/private property runs under trees) with it. It has a reach of around 150-200' and is not suitable for a large project like mine that was 12-14k feet.
It's a great little drill, but I've also spent a fair amount on parts and pieces to keep it running. Nothing too expensive (except the rubber tracks, those are around $500 to replace, did one in 2019 and the other in 2020). I purchased a new locate wand as well, which was over 5k which works better with the beacon than the older one.
If you need to go a short distance it's a good tool, small enough i can move it with a F250.
Interesting, he was able to subsidize nearly 67% of the $126,710 through the 17 pre-orders he received. Probably could have built the entire project with zero out of pocket expenses if it was a more dense area (or at least had a higher signup rate than 30%).
Although, now that it is built what’s the process for hooking up new subscribers? Wouldn’t he have to hire more contractors to bore out lines and hook up the fiber to each home? The cost of acquiring a customer would appear to be astronomical.
- Costs exceeded $126k by far. More like $150k in direct costs plus all the free labor Jared put in. This project has taken literally years of his life and all his spare time lately.
- Pre-orders were more than 17. 17 is the number of subscribers already hooked up.
- Take rate wasn't 30%, it was 70% along the route.
- Customers were, at least partially, pre-installed, so not all will require new construction. Jared also does a lot of the work himself without contractors. He also owns/has access to a chain trencher and a directional bore.
- customer acquisition costs do not traditionally factor in the cost of building drops. These are usually accounted for separately. Obviously construction costs time and/or money.
@jaredmauch Can you share more details on becoming a telco? Did it have other upsides/downsides than easier access to the public right of way? Does being a CLEC have benefits not extended to a BIAS (Broadband Internet Access Provider) as defined by the FCC?
I'm attempting the same thing in red bluff California. Forced to start my own ISP because the other service options suck. Got on a zayo ila site and now it's time to learn how to dig fiber.
Yes and no. If you are living near base station in a big city it will work. But in the countryside where no good connection from ISP equipment to the base station the bottleneck will be this link. Furthermore, 5G uses a higher frequency than LTE, it could be not cost-effective to cover remote regions with small number of users. Starlink could solve problems with cables, but adds problems with RTT. Maybe one day all FAANG companies will open data centres on Earth orbit.
> Maybe one day all FAANG companies will open data centres on Earth orbit.
I know you are joking, but the weight, power, and heat dissipation all make this impractical. They tried similar ideas with cargo containers and underwater enclosures, which were a lot more practical than outer space, but they didn't stick with it.
I think the main problem is spare parts. A long time ago I was working in the data centre. I don't remember any day when there was no problems with hardware.
The signal goes through the ionosphere to the satellite than to another satellite which has a link to the Earth. Starlink uses LEO, so there is 2400 km link(1200 km from/to the ground). 1 ms is time for the signal to travel for 300 km in a vacuum. So theoretically the lowest RTT could be about 8 ms. Right now RTT to my ISP is 2 ms.
Verizon has started rolling out 5G home internet. It's apparently pretty fast - not fiber fast, but still pretty fast, and with truly unlimited data for $70 month. Really exposes just how absolute bullshit their mobile plans at roughly $10/gb are. I've heard good things about it, and looking at their 5G map it's been installed in areas very close to my home in Chicago, but the stations need line of sight for internet and I'm about half a block out of the service area.
Click on a neighborhood and look at the dark red areas running on top of the streets. I have no idea how they're going to get 5g base stations covering all of the smaller residential streets; right now only the bigger streets & downtown are well covered. And even then the internet requires an antenna unit in a window with LoS to the base station, which is OK, but not sure how 5g is going to be practical for phones indoors.
to my knowledge, 5G and other cellular bands are regulated in most countries. You're dependant on an external decision if your area gets served.
If some radio bands with wide range technologies get unregulated for public usage and access to equipment is possible - then communities can decide on their own.
Wireless is a great solution when there is spectrum available. If you have access to low band frequencies as the cellular companies do you can blast through trees, walls, etc.
Even with that they need fiber to their towers to provide the speeds necessary
Yes, there's no suitable spectrum available to towers due to topography, etc..
I might have to build towers at both ends to run in a licensed range, eg: 11Ghz, and even there there's not good access. So I might still have to pay 20-60k to get access on top of the 5-10k for radios and registration of the licensed link. That easily builds 1-1.5 miles of fiber, and since I was just over 2 miles away from it, it became easier to just do that and have something reliable.
He shares his lessons learned (if you feel inclined to build your own) and details his expenses (it's not cheap, but not impossible either). It's a very informative and fun talk!