The simultaneous presence of magnetic fluid and porous structures is relevant to many situations like electronic cooling and thermal regulation using ferrofluids. When convective heat transfer occurs in such applications, the presence of magnetic sources and porous materials induces additional effects that require analysis. The present study addresses passive heat transfer in porous media under the influence of magnetic fields. In particular, the interplay between magnetic and buoyancy-driven convection is addressed, in a hexagonal cavity with differential heating between its side walls and in the presence of permanent magnets. The governing equations are formulated in a dimensionless form and solved numerically using the control volume-based finite element method. The impacts of different parameters, like the gravitational and magnetic Rayleigh numbers, the Darcy number, the thermomagnetic number, and the anisotropy represented by the permeability ratio between two principal axes and the angle between the first principal axis and the horizontal, on the fluid behavior are analyzed. It was found that the magnetic convection acts against or along the gravitational one in the top and bottom of the cavity, respectively. The results show that increasing the magnetic Rayleigh number enhances heat transfer, with a maximum Nusselt number improvement of 60% at high Darcy numbers. Increasing Darcy number can augment Nusselt number by up to 60% at high Rayleigh number. These findings can be beneficial for improving the heating control strategies involving ferrofluids as well in applications where permanent magnets are present.