As a Surface Pattern Designer, both have their place in my workflow. Let me walk you through the difference and how I choose between them.
To me it’s like choosing between making a mosaic and painting a painting. Both are beautiful! but look different when compared to each other even if you make the same motif.

Pixel based art is like working with paint on canvas, you blend the colors and layer your brushstrokes on top of each-other creating subtle color variations and textures.

Vector based art is more like designing a mosaic, you shape your pieces and place them next to each other, but each piece is still a solid color.

Vector designs are made with mathematical points in the design software, I use Adobe Illustrator. The biggest benefit of vector is scalability. You can take your artwork and blow it up to the size of a building, and it will still be perfectly crisp, it never loses quality no matter the size.
It’s also great for clean, precise line-work and easy color editing.

Drawbacks

Drawbacks of Vector-based art, is that it can feel restrictive if you love a more painterly or organic look. I find it very hard achieving natural textures or loose brushwork that looks hand-painted and organic with Vector.
It takes a lot of extra time and effort, so my recommendation for painterly patterns is that pixel based art is probably the better choice, unless you really do need infinite scale-ability which most patterns don’t need.

Raster or Pixel-based art is built from thousands of tiny squares.
Pixel based art gives you more creative freedom (in my opinion). Textures, watercolor effects, layered depth, subtle color variations.
It’s the better choice if you want a pattern that looks hand painted. For work that has subtle color variations and a handmade quality, pixels capture that better.

Drawbacks

Drawbacks of Pixel based patterns is in your resolution. A pixel file is created at a fixed size, and if a client needs it much larger, quality can suffer.
I always ask my clients to think ahead, what size do your manufacturers need to print your textile and work at a high enough resolution for the final use. If you plan accordingly the resolution usually never becomes an issue.

I look at the end use and the look I’m going for.

Solid areas of color, clean shapes, large format print? Vector.

Detailed illustrations with texture, painterly gradients and a handmade feel? Pixels.

But no matter which I choose, I always sketch by hand first. Something about paper gives me freer lines and more creative ideas. The digital work comes after.

So in conclusion, There’s no right answer here. Both work beautifully in pattern design.
Personally? Photoshop is my favorite. I’ve been using it since I was a teenager and it’s where I feel most at home. It’s my default software in most cases.

Learning Illustrator has been a great addition to my toolkit though, and sometimes vector just makes more sense for the job.