Few investments have outperformed Apple (Nasdaq: AAPL), yet for a select group of well-heeled investors there is just as much allure in owning, say, a Paul Cezanne painting of the fruit. Which is to say that as the world’s population of high-net-worth investors continues to grow there has been a correlated explosion in investment in the art world.

The past decade has proven flush for art investing. According to the Mei Moses All Art Index art returned a robust 10.2% in 2011, while the Standard & Poor’s 500 finished flat. Over time such returns have been the rule, according to research compiled by the art market research firms Deloitte Luxembourg and Artastic. According to Deloitte from 2000 through 2011 post-war and contemporary art—including such artists as Jackson Pollock, Roy Lichtenstein and Andy Warhol—as a class returned 11.4%, and traditional Chinese art returned 15.8%, while overall markets eked out gains of 0.5%. In total, global art markets have grown from $21 billion in 2002, to $64.9 billion at the end of 2011.

In short, the art market remains not only cool to own, but hot. “If you look at all indices that compare art versus other investments, the art market has done well,” says Michael Plummer, principal and co-founder of Artvest Partners, a firm that advises wealthy clients on how to invest in art. “When you cross-check the data the analysis holds up.”

Certainly high-priced art has been in the public eye recently. Edvard Munch’s The Scream recently sold at auction for $119.2 million, and on May 8 a 1961 painting by Mark Rothko set a record for post-war art, selling for $87 million.

Yet amidst the excitement art remains a dicey place to seek alpha. For one thing, commissions are enormous, consuming up to 40% of the selling price of a piece, says Plummer. Knowing this, he advises clients to hold pieces for at least three years, with five being the more traditional standard.

And not everyone is convinced the performance data for art is meaningful. For starters, the Mei Moses index only uses data gathered from pieces of art that have sold at auction twice, says Amy Goldrich, an adjunct instructor for the “Wealth Management In The Art Market” course at New York University, and an art attorney at Lynn & Cahill, llp in New York. “The index excludes things at auction that don’t sell,” she says. This leads to the issue of survivor bias in the index, where pieces that are unwanted don’t impact its tally, skewing results.

(Some in the industry are trying to build a better system of tracking art’s value to bring transparency to the market.)

Goldrich also noted that auctions represent just a fraction of all transactions in the art market. Many high-end works only trade hands privately. How many? She wouldn’t guess, although by some estimations auctions may only account for perhaps 50% of the total global market.

Art investors also have special tax considerations. The Internal Revenue Service has a distinct categorization for an art investor versus someone who is merely a collector. Both will get taxed at the capital gains level for any profits made on works sold, but only the investor will be able to make a series of deductions, including the cost of travel to art fairs, says Goldrich. “People who are seen as more serious collectors say they buy art, they don’t sell it,” she says. “These are all things someone who is not an investor would say.”

Here’s another landmine, when art is passed along to heirs its value has to be declared along with all other possessions of the official estate. If the art is not, and the recipient later wants to sell the piece any reputable auction house will ask for documentation. If it can’t be proven that the appropriate estate taxes were paid, the auction house won’t sell the work.

Another problem is the art market is ruled by taste alone, as its assets produce no cash, and have no inherent value. As such art markets are notoriously fickle. Goldrich points to 1980s art kingpin Julian Schnabel as one example of a big name that hasn’t worn well over the following decades. Also, entire segments of the art world can become old-hat. Artvest’s Plummer notes that American art from before 1950 did very well until global financial markets crashed in 2007, and this sector hasn’t rallied since.

Even some advisors who help their clients buy art are skeptical. “I’m not a big believer in art as an investment,” says Natasha Pearl, CEO of Aston Pearl, an advisor to single family offices. Pearl estimates art-related assignments make up 25% of her business. “There is no Blue Book, this is not like buying a used car.” What Pearl means is that the art market is highly illiquid, and filled with unique pieces that make it hard to reach consensus on value. As such, it is easy to buy a big, expensive, prestigious dud. “The name of the artist is not enough,” she says. “You have to look at provenance, past sales history, and project out what other factors can influence price.”

Across the board the people I spoke to said that would-be art investors need to use the services of qualified expert before they make any major purchases. This person not only knows the ins and outs of the particular part of the market you are interested in, but works for you, not the seller. They can not only help you know the right price for a piece, but ensure that you aren’t about to buy an ill-documented fake.

The singularity of art is part of what makes it so appealing, but also so dangerous. For your investment, which could easily cost tens of thousands of dollars you can only buy, really, one piece. There is no investible index for the art world that you can own for small money, at least yet. And while there are art-based investment funds these are not for the faint-of-heart either. The London-based Fine Art Fund Group, for example, has a minimum buy-in of $250,000, and its managers are compensated to the tune off 1-3% of the annual net asset value of the portfolio, and 20% of the profits on all art sold. Which sounds a lot like a hedge fund.

If you are determined to invest in art, there are a few red flags to avoid, says Christian Trabue supervising appraiser and head of the fine arts department at Enservio Select, a personal property appraisal firm. She cautions that newer artists, as compared to established masters, are considered high risk stocks in her field, as they have little in the way of track records.

Trabue also cautions against lithographs and etchings, as these can be endlessly reproduced, and there is a history of fraud in this part of the market. “If I’m going to spend thousands of dollars, I’m going to go for the original over the print,” she says.

Another issue is that there is no certification required to own or work in a gallery. And there is no art world version of the Securities and Exchange Commission, ineffectual as it may be.

A final caution for investors would be that they stay aware of their own need for social approval, and status when they enter a gallery or auction house, says NYU’s Goldrich. The art world preys upon the insecure wealthy. “People who are brilliant take leave of their senses,” she says. “There is a neediness that comes out, where they want to be viewed not just as a money person, but as a tasteful person. But you can’t buy taste.”