The confusing terminology mentioned in the question deals with entering and exiting option orders. In review, there are two main ways in which you can participate in options: You can either buy an option or write an option (also called selling an option). When you buy an option you are purchasing the right to either buy (call option) or sell (put option) the underlying asset at a set price before a set date. If you write an option, you are selling this right for a premium.
When you enter a trade, you are essentially opening a position, hence the phrases “sell to open” and “buy to open.” If you are buying an option, either a put or a call, you must enter a “buy to open” order. If you are writing an option, you must enter a “sell to open” order.
Now, to exit an order, you need to close your options position. If you bought an option, you need to use a “sell to close” order, which is essentially like owning a stock where you sell it back into the market to close out the position. If you wrote an option, you will need to use the “buy to close” order. While it may seem odd that you would buy to close a position, by taking a long position in the option, you neutralize the rights you sold when you wrote the option with the rights gained when you buy the new options, which closes your position.
For example, let’s look at a call option on Citigroup (C) stock, trading for a $1.45 premium and set to expire two-and-a-half months from now on Jan.19, 2018. Citi’s stock price is currently trading for $74 and the strike price on the call is $78. A trader who wants to buy the rights to purchase Citi in two and-a-half months time for $78 may decide to buy this call option. When he makes the order through his brokerage account, he’ll enter a buy to open order, thereby opening his position on the option. If Citigroup’s stock price increases to, say, $80.00 before it expires, the trader will exercise his option with a sell to close order. This means that his open option will be closed when he sells the option.
In summary, a person holding a short position (contract writer) can sell to open (enter a contract) or buy to close (close a position). A person holding a long position (contract purchaser) can buy to open (enter a position) or sell to close (close a position).